General game developing news and updates.
Play The Elder Scrolls V: Skyrim Special Edition for FREE starting now through Sunday at 1PM Pacific Time. You can also pickup The Elder Scrolls V: Skyrim Special Edition at 50% off the regular price!*
If you already have Steam installed, click here to install or play The Elder Scrolls V: Skyrim Special Edition. If you don’t have Steam, you can download it here.
*Offer ends Monday at 10AM Pacific Time
Hearthstone lead developer Ben Brode has some interesting thoughts on how the popular card-battler manages to balance player agency and strategy with randomness.
Speaking to TouchArcade during a recent interview, Brode explaines the genre is driven by the notion of randomness, and the idea that players are continuously thrust into new and unfamiliar situations.
In card-battlers, for instance, each deck is usually shuffled before each round. While the result of that shuffle might be considered lucky or unlucky depending on the outcome, the best players will be able defy the odds by thinking outside the box and cooking up new strategies.
“Hearthstone is much less random than other card games. Having that guaranteed mana each round means we have to go and look for ways to keep it from each match being too similar,” says Brode.
“We don’t want there to be one ideal class or deck that just wins the majority of the matches so we spend a lot of time looking for ways to keep matches fresh. We want you as a player to find yourself in situations that require you to think on the fly.
“One of the things people often get wrong is that they say this game is luck based or skill based. Games don’t really work like that though. Tic-tac-toe is low skill and low luck whereas poker is high skill and high luck. It’s not a slider that you go one direction or the other on.”
He suggests Hearthstone is extremely similar to poker in that regard, positing that the game’s high skill requirement exists because of its random events and abilities. He believes the skill required to succeed increases when the game becomes more diverse, raising the stakes and pushing players to make the best plays possible.
You can hear more from Brode by checking out the full interview over on TouchArcade.
The developers at Eugen Systems have ceased their strike after negotiations with management came to a standstill.
It’s an acrimonious end to proceedings, with those on strike deciding to conserve their resources for future endeavors after management refused to budge.
The strike began back in February, when the Wargame and Steel Division maker saw 21 of its 44 employees walk out in protest over unfair treatment.
The studio was lambasted by staff for failing to compensate them for overtime, ignoring minimum wage laws, and refusing to acknowledge employee contracts.
The strike lasted almost two months, but those protesting admitted they were struggling to make any headway, having hit out at management just two weeks ago for ignoring their demands and refusing to negotiate.
“After six weeks on strike and an obvious lack of will to change the situation, Eugen Systems’ management finally show their true colors and aren’t trying to hide the fact they have no intention negotiating anymore,” wrote Eugen’s striking workers towards the end of March.
“Our last meeting of Friday, March 23, lasted only a quarter of an hour. And for good reason, confronted with systematic rejection from management, we asked whether they would accept to negotiate anything. Their answer was brief and precise: no.”
The Eugen management team clearly hasn’t experienced a change of heart in the two weeks since, but even though the strike has petered out, the company’s employees claim the battle is far from over.
“We will continue to fight for our rights with the legal means at our disposal. Therefore, approximately fifteen Eugen Systems employees and ex-employees have seized the prud’hommes (French labor tribunal),” they wrote in a new post on the Union of Video Game Workers website.
“It is thanks to the support we received that we were able to hold out for more than a month and a half. The public interest (media, politicians, players…) for this novel social movement reinforces us in the idea that it was not in vain, and that we were right to fight for our rights. We want this industry to mature, to recognize the value of our work and of our skills.”
Veteran animator and animation director Kristjan Zadziuk, whose credits include Splinter Cell: Blacklist and Watch Dogs 2, has posted the first in a new video series of gameplay breakdowns dissecting the mechanics behind popular titles.
The subject for Zadziuk’s debut video is Sony Santa Monica’s impressive-looking God of War reboot, with the former Ubisoft man keen to explore how the studio created some of the game’s wonderfully fluid combat animations.
Throughout the video, Zadziuk serves up an wonderfully detailed deconstruction of one of Kratos’ new movesets, analyzing everything from how the camera reacts to the God of War’s movements through to the transition animations that glue everything together.
“Here is where we see the entry into the defense stance,” he explains, referring to the moment Kratos deploys his new retractable shield. “They’ve been really smart here, and instead of creating a ton of transition animations to do this stance, they’ve basically let the blend do the work.
“You can see here, it looks like it’s about a three to five frame full-body blend, and when you look closely you can actually see where this transition happens.”
Naturally, Zadziuk’s commentary works best when its accompanied by the reference material in the video itself, so be sure to give it a watch if you’re in the market for some rather insightful game dev tidbits.
Failbetter Games, that far-off British emporium of text-driven tales, made waves a few years ago with the release of Sunless Seas, a horror-themed storytelling game that wove its way through Early Access before filling players with terror, dread, and arousement as a complete package.
This year, Sunless Skies is making the same journey, taking the Failbetter world from the ocean’s depths into the void of space. It’s still in Early Access, but that makes it the perfect opportunity to chat with the folks at Failbetter about how things are going in development and asking what they’ve learned, and what they think they have yet to learn.
So today at 1PM EDT (2 hours earlier than our usual jaunts), we’re going to be chatting with QA lead Lesleyann White while streaming Sunless Skies for your terror and amusement. Be sure to join us and ask your questions!
And while you’re at it, be sure to follow the Gamasutra Twitch channel for more developer interviews, editor roundtables, and gameplay commentary.
The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.
Crossposted from Sharkbomb Studios blog.
For most people Early Access is synonymous with Steam. But with me being a solo-developer and Steam being such a big and important marketplace, I did not want to start with my game there. So instead, I launched Nowhere Prophet on itch.io.
Now, 6 months later I can say it was the right decision. So far I’ve made more than $10.000 in gross revenue on and a noticeable amount of that is from voluntary tips and the kickstarter-like rewards I offer, things that would not have been possible on Steam.
However Steam still plays a role, even when you’re selling elsewhere. Nowhere Prophet has been on Steam ever since the death of greenlight and a visible chunk of traffic and purchases comes from Steam, but more importantly: The sales spikes on itch.io correlate with increased wishlist additions, a key factor for launch success on Steam.
So, read on to find out what I did and how it all shakes out. Including all the interesting numbers for Nowhere Prophet. And as a little bonus: I’ve got some sales data from other itch.io titles for comparison, so you can look forward to even more graphs!
Nowhere Prophet is a roguelike deck-building game that I’ve started working on in May of 2014. It’s been a long and winding road but since the 10th of October 2017 the game has been available in what I’ve dubbed First Access on the indie game store itch.io.
[embedded content]
It’s Early Access and Crowdfunding in one. That means you can buy and play the game while I’m still developing it. That’s the Early Access part. However you can also spend even more money than the $19.99 price tag to have something of yours (like your name, your face etc.) become a part of the game.
I’ve decided to run a First Access phase for three key reasons:
1. My Happiness
I’m a solo developer. That means I’ve been working on Nowhere Prophet mostly by myself. Granted there are a bunch of people helping out here and there but all in all it’s me against the world. And I’ve been doing that for long enough that development had become draining. I just knew I had to make the game ready for actual players. I had to get it out there. To see how it is received.
2. Their Feedback
I want to make this game as good as I can and that means I need some help. Again, working on it by myself means I have only one constant point of view on the systems and content. There’s no back and forth within a team on the daily issues. And that’s not good enough. So another goal was to get some actually player feedback to inform the design going forward.
3. Our Community
And lastly I wanted to start building a community around the game. Get people involved and excited. After all it’s consistently been one of the most rewarding things to see folks play the game at conventions so a lively community sounds like fun. It’s also helpful for spreading the word and making the launch a success, I hope.
I wrote an article on the first few weeks of the First Access, which also explains my reasons for going with itch over Steam. But now, let’s get into the numbers.
Let’s start with the most obvious graph, the sales of the last six months.
Note that these amounts (and all others, unless otherwise specified) are gross revenue. That is they are the money paid by the buyers to itch, without any deductions in place yet. The money that ends up in my hands is less than that. I’ll go over how this breaks down in a bit, but first, let’s look at the graph:
As you can see there’s a soft upwards movement in the valleys over time (until recently), but clearly more interesting are those spikes that are in there.
Let’s do some detective work on those dates.
The first one is obviously the launch spike, right at the gate. The second spike coincides with SplatterCatGaming’s two YouTube Videos showing off Nowhere Prophet. The one towards the end is from one backer that purchased the 350$ THE TALE option. And then there’s the towering spike in the middle of the graph, and that one’s clearly from the excellent Rock Paper Shotgun preview. There also is a slightly elevated plateau from when Pladd started streaming the game on Twitch (correctly named Follower Death Simulator 2018 on his streams) but that’s hard to pick out in the graph above.
Breaking down the impact of these sales periods is hard to do precisely, but here’s my best guess.
First let’s look at a stretch without any spikes: The 70 days from 4th November to 12th of January. During that time period the sales stayed more or less level.
And these 70 days give us an average of revenue per day ($23.76). That’s going to be our baseline. Next we look at the general spike duration (~8 days) and track the revenue for each of the spikes for that time. And that gets us:
And lastly, we can do something pretty useless, but funny: Compare the “attributable revenue” from that spike with the number of views, followers or subscriptions. Let’s take a look at that.
SplatterCatGaming (YouTube)
SplatterCatGaming has 440,000 subscribers and both videos have a total of 95.141 views (82,391 for the first plus 12,750 for the second). Of course these are the numbers for now, not the numbers right after the 8 day period, but this is more for fun than anything really useful.
Rock Paper Shotgun
To measure the “reach” of Rock Paper Shotgun I took a look at their Facebook page likes (98,505) and Twitter followers (191,000). Obviously these aren’t the best way to estimate the visits on the article but it’s something.
Pladd (Twitch)
Pladd has a total of 3,120 followers on twitch. That makes for a whopping:
Some more thoughts:
My game updates don’t seem to correlate with any spikes. I’ve heard from other teams that big updates can actually create a visible push in the sales numbers, even without coverage, but that is probably due to those team having good reach and more time to advertise their changes.
Also I think the active development is probably what contributes to that soft rise over time. However the recent drop off is harder to pin down – maybe I’ve reached saturation on itch?
Another thing I can do with the data is to look at how the purchases and revenue are spread across different countries. In total there’s purchases from 46 different countries.
It’s not surprising that the US makes up the bulk of the purchases. Germany comes second, probably because I’m a German developer and I’m well connected here. That’s followed by Great Britain, the Netherlands, Canada and then France.
If we split it by continent, with Canada and the US lumped into their own thing, the graph becomes a bit easier to read. And you can see, unsurprisingly that the majority of the sales come from North America and Europe.
Alright, now for the key thing that’s different from regular First Access, and one of my main reasons for Itch: The Rewards!
Considering I invested about 2 days to design, plan, get feedback on the tier rewards, and about a day to set up the page, make the graphics, write the descriptions, we come out at $290,11 per day spent on the tiers. That’s decent.
But that does not include the time spent to communicate with backers nor the time spent working on the actual content. So this number again, is more fun than actually useful.
However what is maybe slightly more interesting:
I have reached out to all my 38 reward-level backers via e-mail, some of them multiple times, but so far the amount of replies (and thus the implementation and fulfillment) is just above 50%. That means almost half of all reward-tier backers never reached out to me with the details of their reward. This includes a few of the people who paid $80 for the THE FACE reward.
I’m still a bit baffled by this, but not overly surprised, since this seems to be the case for many kickstarters as well. One would think that this might be friends and family supporting the game but not wanting the reward but that doesn’t seem to be the case. All the non-responsive backers are e-mails I do not recognize.
And one final thing on the revenue: Itch.io allows buyers to tip the developer, by setting a custom price themselves. This means there’s three sources of revenue: The base price, the difference to a reward price an the tips. Here’s how those break down:
What I found surprising in that is the sheer amount of tips. I have made some experiments with donations in games in the past but with very little success. It feels as if the itch.io buyer community is very much pro indie and willing to support developers with a little extra, if they can. It’s much appreciated.
Another interesting set of data is related to Steam. The Nowhere Prophet store page and the community page have been available on Steam since Mid 2017, about half a year prior to the Early Access.
During that time it had accumulated almost 500 wishlist additions without being very active there. That’s neat. But what’s even more interesting is if we take the wishlist additions (and deletions) during the First Access period and compare them to the itch.io purchases:
As you can see these two are linked! That means that when the game’s being bought during spikes, those people that don’t buy it probably wishlist it on Steam. That’s good to know. Especially since wishlists seem to be a good indicator for Steam launch success.
So, and as mentioned earlier all the numbers given so far are gross revenue. That is they are what comes in to itch. That’s not what actually ends up on my end. From the revenue you have to deduct VAT (where added, not all countries), payment provider fees and of course the itch store fees. The result are the net earnings.
For example, you sell something for $20. Based on that amount the itch fees and payment fees (based on the payment provider used by the buyer) are set. In countries where VAT (the height of which is based on the country of the buyer) is added, that is put on top.
Now depending on the payment model you selected (direct to you, or via dedicated payouts), Itch can either pay the VAT for you, deducting it from your revenue, or you pay VAT yourself. Regardless, what remains are your earnings.
With my share for itch set to 10%, the default, I come out with about 85% of the revenue. On Steam, with the platform share set to 30% and the VAT (again, where applicable) being deducted from the sale price (not put on top as with itch) that share will turn out to be around 65%.
Total Revenue
So, with all these numbers, where do I stand currently in regards to the moolah?
That does look good. It certainly is better than I had expected. Granted I have set my expectation purposefully low but still, this is encouraging when it comes to a full launch on steam.
However these figures need some context to really make sense. So let’s compare them to other games on itch, and to the estimated Nowhere Prophet the development cost.
Now what’s interesting is to see how all this that compares to other games sold on itch. Let’s look at Overland, Cultist Simulator, Solitune and two more anonymous games which were all generous enough to share their data with me.
At first glance it looks like Overland is a lot more successful, which isn’t really surprising – it’s been generating a good amount of buzz. However Overland has been available for two years now. This data gets a bit more interesting if you break that down to a graph of gross revenue per day since launch. Ideally I’d have compared the first six months for each game but that’d require appropriately detailed data.
Anyway, here you go:
That gives an interesting perspective and a surprisingly wide range. However what’s less surprising that even in this small cross-section you can already see how only a few games are really successful and then it quickly dips down. It’s the same in Steam and probably on any other digital marketplace.
However with that said, Nowhere Prophet seems to be in a good place. That’s reassuring.
Also, here’s some info on the games, to provide additional context.
Solitune
Launched in November 2016, Solitune is a small playable experience sold for $2 a piece. It’s also available on Steam.
Anoynmous 1
This game was also launched without Early Access in late 2017. It’s been available on a variety of platforms and interestingly their Itch.io sales amount to about 12% of total revenue.
Anonymous 2
This game’s Early Access was made available exclusively on itch starting mid 2017. It’s comparable to Nowhere Prophet in terms of genre and price. As a small note: The price of the game has varied over time.
Cultist Simulator
Launched in October 2017 it was available for an Early Access sale until end of February. Interestingly that creates a solid sales peak on the last days it was available. You can find out more about the numbers in this article.
Overland
Launched its First Access in the middle of April 2016, it’s probably one of the more successful titles. It’s not available elsewhere. Overland is an interesting data point because it’s a similar game and price point to Nowhere Prophet.
One point of note is that none of these games have offered kickstarter-like rewards and not all of them had been in Early Access.
Now it would be interesting to compare these figures to the development cost. Unfortunately that’s not something I can do for the other games, but it’s something I can do for Nowhere Prophet!
And something else we can do with the total earnings is: We can hold them against the development costs and do a projection for Steam.
Quoting Alexis Kennedy who roughly and anecdotally assumes itch to make up about 3% of the final Steam sales. That feels a tad optimistic, but it’s as good a number as any. So with that applied I could reasonably sell about 15,000 Units in a similar time period on Steam.
So where does that put me in regards to break even? To answer that we first have to figure out how much development has and will cost.
For my freelancers and assorted costs I have invoices that I can put down and that amount to about $65,000. But for my own (mostly unpaid work) it is a little trickier.
Luckily I have tracked the time I’ve spent on the game. And that comes out at a total of – oh god – 4275 hours, or about 534 working days so far. Since there’s going to be a few more months of hard work ahead, let’s just round up to a full 600 days of work and let’s put a daily rate of $300. That gets us to $180,000 for 3-4 years of development time. That wouldn’t be what you’d call a stellar yearly salary but what are you gonna do.
So add these two things together and the game will have cost about $245,000. Let’s just round that up to $250.000, a cool quarter mil.
(Note: I’m based in Germany so all my internal calculations are in Euro. To make this properly line up with US Dollars I’m assuming a conversion rate of $1.2 to 1.0€)
So?
With all these numbers laid out, how am I faring?
Let’s say I’ll get some more sales on itch to amount to $10,000 in earnings, that leaves a hole of $240,000. So how much would I have to sell on Steam to hit that?
Assuming a $19.99 price and 65% profit from that, that means $13 per unit sold. But wait, most people on Steam will probably buy during a sale – that’s the reality, so let’s take off another 20% so it’s more like $10 per unit.
That means I’d have to sell about 24,000 copies on Steam. That’s a good lot more than the currently projected 15,000, so… Uh… I guess that means I’ll have to pump those rookie numbers up and hope for some good coverage from press and influencers.
Right, so that was a long and rambling article. What can I take away from all these numbers?
Success Story?
Firstly, after six months of First Access, did it meet my goals?
Yes, yes and no.
I’m definitely happier now that the game’s been out in the wild. It’s great seeing people interact, watching streamers engage with the game and seeing people share the game in reddit thread. Going to Discord also was a great decision. My forum is still pretty inactive and most of the activity is in the Discord chat.
Feedback has also been great. I have a few engaged players that help me fine tune the game and it’s been great hearing what people like and not like. For example I’ve started focusing more on adding depth to the unlockables, because I’ve found out that people want some more meat there.
However I have less of a community than I had hoped for. There are a few very active players that provide feedback and ideas, and there are bunch more fans that idle in the channel and that show up around the larger updates. However what it comes down to is: There’s not really any activity unless I’m around. And that’s unfortunately not quite what I would call a healthy community.
Surprising Surges!
Sales have been higher than expected. The itch.io featuring around launch and the game showing up at a good spot in the roguelike tag of the store are a constant source of visits. Also the spikes from the streamers that have picked up the game unprompted and the RPS article have been great.
However these have also been random. I have reached out to many press outlets around launch but with very little success. Almost all the coverage I have gotten has been chance encounters or them discovering the game. That’s not really heartening going forward, but that’s how I’ve experienced things so far. Granted I might just be bad at reaching out.
Small Sources.
One interesting thing is how consistently a few reddit threads and the steam page brings in traffic. It’s not a lot – just about 100 hits each month in total. But that’s good. Compared with the wishlist surges it probably pays to have the Steam page up as early as possible. Also note that itch shows me the traffic source for the purchases and I can tie 14 out of all 450 purchases to the Steam page. That’s neat.
Also the reddit threads are especially interesting since these are examples of fans sharing the game in an appropriate post. That’s been super motivating for me and since a few of these posts are in a slightly slow-moving subreddit (r/roguelikes) they’ve been bringing in traffic for a while now.
Well, I keep working on the game, of course. I just detailed the roadmap for Q2 the other day (you can read more here) and I hope to have it in a final shape this year.
And that’s all I’ve got for now.
Thanks for reading and if you’ve got any questions, do ask! You can reach me on twitter (@mnerurkar). Also if you’re interested in digital card games, maybe go check out Nowhere Prophet, buy it on itch.io or just wishlist the game on Steam.
© 2018 Valve Corporation. All rights reserved. All trademarks are property of their respective owners in the US and other countries.
View mobile website
© 2018 Valve Corporation. All rights reserved. All trademarks are property of their respective owners in the US and other countries.
View mobile website
7.13b:
==
* Strength heroes base movement speed reduced by 5
* Strength heroes base armor reduced by 1 (except Doom, Io, Lycan, Phoenix, Timbersaw, Tiny)
* Intelligence heroes base strength increased by 2 (Except OD, Tinker and Bane)
* Strength gain for the following heroes increased by 0.3: Dark Seer, Death Prophet, Ember Spirit, Leshrac, Nature’s Prophet, Necrophos, Pugna, Queen of Pain, Ursa and Visage
Since the release of the Dreamcast and the modem adapter, game developers have been able to collect data from players about their behavior in the wild. Game analytics actually goes further back when considering early online PC titles such as EverQuest, which was released in 1999. Game servers were necessary for authenticating users and population game worlds, but also provided the capability to record data about gameplay.
Since 1999, the landscape for collecting and analyzing data has changed significantly. Rather than storing data locally via log files, modern systems can track activity and apply machine learning in near real-time. Here’s the four stages of game analytics systems I’ve noticed during my tenure in the game industry:
Each of the steps in this evolution support the collection of larger data sets, and reduce the latency from gathering data to performing analysis. In this post, I’ll introduce example systems from each of these eras, and discuss pros and cons of each approach.
Game analytics really started gaining momentum around 2009. At Bioware, Georg Zoeller built a system for collecting game telemetry during development. He presented the system at GDC 2010. Shortly after, Electronic Arts started collecting data from games post development, to track player behavior in the wild. There was also growing academic interest in applying analysis to game telemetry. Researchers in this field, such as Ben Medler, proposed using game analytics to personalize experiences.
While there has been a general evolution of gameplay analytics pipelines over the past two decades, there’s not a fixed timeline for the progression between the different eras. Some game teams are still using systems from the earlier eras, and it may be the best fit for their use cases. There’s also a number of vendor solutions that are available for game analytics, but I won’t cover those in this post. I’m focusing on game teams that want to collect gameplay telemetry and own the data pipeline being used.
Components in a pre-database Analytics Architecture
I got started in game analytics at Electronic Arts in 2010, before EA had an organization built around data. While many game companies were already collecting massive amounts of data about gameplay, most telemetry was stored in the form of log files or other flat file formats that were stored locally on the game servers. Nothing could be queried directly, and calculating basic metrics such as monthly active users (MAU) took substantial effort.
At Electronic Arts, a replay feature was built into Madden NFL 11 which provided an unexpected source of game telemetry. After every game, a game summary in an XML format was sent to a game server that listed each play called, moves taken during the play, and the result of the down. This resulted in millions of files that could be analyzed to learn more about how players interacted with Madden football in the wild. During my internship at EA in fall 2010, I build a regression model which analyzed which features were most influential in driving player retention.
The impact of win rates on player retention in Madden NFL 11 based on preferred game mode.
About a decade before I started my internship at EA, Sony Online Entertainment was already using game analytics, by collecting gameplay data via log files stored on servers. It wasn’t until a few years later that these data sets were used for analysis and modeling, but it was still one of the first examples of game analytics. Researchers including Dmitri Williams and Nick Yee published papers based on data analyzed from the EverQuest franchise.
Storing data locally is by far the easiest approach to take when collecting gameplay data. For example, I wrote a tutorial on using PHP to store data generated by Infinite Mario. But this approach does have significant drawbacks. Here’s an overview of the tradeoffs with the approach:
Pros
– Simple: save whatever data you want, in whatever format you want
Cons
– No fault tolerance
– Data is not stored in a central location
– Huge latency in data availability
– No standard tooling or ecosystem for analysis
Flat files can work fine if you only have a few servers, but it’s not really a analytics pipeline unless you move the files to a central location. At EA, I wrote a script to pull XML files from dozens of servers to a single server that parsed the files and stored the game events in a Postgres database. This meant that we could perform analysis on gameplay data for Madden, but the dataset was incomplete and had significant latency. It was a precursor to the next era of game analytics.
Another approach that was used during this era was scrapping web sites to collect gameplay data for analysis. During my graduate research, I scrapped websites such as TeamLiquid and GosuGamers to build a collection of professional StarCraft replays. I then build a predictive model for identifying build orders. Other types of analytics projects during this era include scrapping websites such as the WoW Armory, and more recently SteamSpy.
Components in an ETL-based Analytics Architecture
The utility of collecting game telemetry in a central location became apparent around 2010, and many game companies started saving game telemetry in databases. A number of different approaches were used to get event data into a database for analysts to use.
While I was at Sony Online Entertainment, we had game servers save event files to a central file server every couple of minutes. The file server then ran an ETL process about once an hour that fast loaded these event files into our analytics database, which was Vertica at the time. This process had a reasonable latency, about one hour from a game client sending an event to the data being queryable in our analytics database. It also scaled to a large volume of data, but required using a fixed schema for event data.
When I was a Twitch, we used a similar process for one of our analytics databases. The main difference from the approach at SOE was that instead of having game servers scp files to a central location, we used Amazon Kinesis to stream events from servers to a staging area on S3. We then used an ETL process to fast load data into Redshift for analysis. Since then, Twitch has shifted to a data lake approach, in order to scale to a larger volume of data and to provide more options for querying the datasets.
The databases used at SOE and Twitch were immensely valuable for both of the companies, but we did run into challenges as we scaled the amount of data stored. As we collected more detailed information about gameplay, we could no longer keep complete event history in our tables and needed to truncate data older than a few months. This is fine if you can set up summary tables that maintain the most important details about these events, but it’s not an ideal situation.
One of the issues with this approach is that the staging server becomes a central point of failure. It’s also possible for bottlenecks to arise where one game sends way too many events, causing events to be dropped across all of the titles. Another issue is query performance as you scale up the number of analysts working with the database. A team of a few analysts working with a few months of gameplay data may work fine, but after collecting years of data and growing the number of analysts, query performance can be a significant problem, causing some queries to take hours to complete.
Pros
– All data is stored in one place and is queryable with SQL
– Good tooling available, such as Tableau and DataGrip
Cons
– It’s expensive to keep all data in a database like Vertica or Redshift
– Events need to have a fixed schema
– Truncating tables may be necessary
Another issue with using a database as the main interface for gameplay data is that machine learning tools such as Spark’s MLlib cannot be used effectively, since the relevant data needs to be unloaded from the database before it can be operated on. One of the ways of overcoming this limitation is to store gameplay data in a format and storage layer that works well with Big Data tools, such as saving events as Parquet files on S3. This type of configuration became more population in the next era, and gets around the limitations of needed to truncate tables and the reduces the cost of keeping all data.
Components in a Data Lake Analytics Architecture
The data storage pattern that was most common while I was working at a data scientist in the game industry was a data lake pattern. The general pattern is to store semi-structured data in a distributed database, and run ETL processes to extract the most relevant data to analytics databases. A number of different tools can be used for the distributed database: at Electronic Arts we used Hadoop, at Microsoft Studios we used Cosmos, and at Twitch we used S3.
This approach enables teams to scale to massive volumes of data, and provides additional fault tolerance. The main downside is that it introduces additional complexity, and can result in analysts having access to less data than if a traditional database approach was used, due to lack of tooling or access policies. Most analysts will interact with data in the same way in this model, using an analytics database populated from data lake ETLs.
One of the benefits of this approach is that it supports a variety of different event schemas, and you can change the attributes of an event without impacting the analytics database. Another advantage is that analytics teams can use tools such as Spark SQL to work with the data lake directly. However, most places I worked at restricted access to the data lake, eliminating many of the benefits of this model.
Pros
– Scales to massive amounts of data
– Supports flexible event schemas
– Expensive queries can be migrated to the data lake
Cons
– Significant operational overhead
– ETL processes may introduce significant latency
– Some data lakes lack mature tooling
The main drawback with the data lake approach is that usually a whole team is needed just to keep the system operational. This makes sense for large organizations, but may be overkill for smaller companies. One of the ways of taking advantage of using a data lake without the cost of operational overhead is by using managed services.
Components in a managed Analytics Architecture (GCP)
In the current era, game analytics platforms incorporate a number of managed services, which enable teams to work with data in near real-time, scale up systems as necessary, and reduce the overhead of maintaining servers. I never experienced this era while I was working in the game industry, but saw signs of this transition happening. Riot Games is using Spark for ETL processes and machine learning, and needed to spin up infrastructure on demand. Some game teams are using elastic computing methods for game services, and it makes sense to utilize this approach for analytics as well.
After GDC 2018, I decided to try out building a sample pipeline. In my current job I’ve been using Google Cloud Platform, and it seems to have good tooling for setting up a managed data lake and query environment. The result was this tutorial, which uses DataFlow to build a scalable pipeline.
Pros
– The same benefits as using a data lake
– Autoscales based on storage and query needs
– Minimal operational overhead
Cons
– Managed services can be expensive
– Many services are platform specific and may not be portable
In my career I had the most success working with the database era approach, since it provided the analytics team with access to all of the relevant data. However, it wasn’t a setup that would continue to scale and most teams that I worked on have since moved to data lake environments. In order for a data lake environment to be successful, analysts teams need access to the underlying data, and mature tooling to support their processes. If I were to build a pipeline today, I would definitely start with a serverless approach.
