Andrew Waag, Manager of Data Analytics at NBC Universal and Victor Wang, Senior Product Manager Mobile Games at NBC Universal joins Melissa Zeloof from ironSource and Joe Kim from GameMakers.com on LevelUp: Game Product Managers Edition. Tune in to hear them talk about best practices for building data infrastructures for free-to-play games, tips for setting up the right KPIs, and common mistakes in game analyses.
Read on for edited highlights from the podcast:
Defining KPIs
“KPIs stand for key performance indicators. If they aren’t key to your understanding of the game, they shouldn’t even be considered KPIs to begin with.”
The three sets of KPIs
“Overall, KPIs are important to understand three different things about your game: retention, engagement, and monetization.
The most essential set of KPIs serves the purpose of understanding business performance, which is usually of most interest to executive-level management. Ultimately, these are KPIs that help game teams understand the relationship between LTV and CPI. This way, you know as early as possible that you have an ROI+ game. Under LTV, you can figure out what your retention curve looks like. Looking at that retention curve, you can determine the average retention days, which is the area below that curve. You can also look at ARPDAU. On the CPI side, you can either look at what your CPIs are by network or by campaign.
The next set of KPIs goes a level deeper in unpacking LTV, and is something product teams are interested in knowing in order to monitor and optimize their games. For example DAU, which is a function of installs and retention curve. ARPDAU can be unpacked into player conversion rate, transaction value, number of payments per pair.
Then one level deeper is to understand in-game player behavior, something that game economists and designers are interested in knowing. On the engagement side, you have KPIs like session length, number of sessions, participation rates of certain events and features. On the monetization side, you have KPIs like currency inflow and outflow by certain contexts within the game, and conversion rate by actual package.”
Creating custom metrics
“Industry standard metrics give you the advantage of being able to compare your product to others. As organizations mature, they tend to standardize their own metrics that serve their needs better. In almost all cases, there’s a need to have custom metrics. Perhaps games without complex economies or varied gameplay, like hyper-casual, could get by with a standard set of metrics like retention, time spent in-game, ad impressions per DAU, etc.
Ultimately, product teams need to be asking themselves, what are the things that need to happen in my product for it to be a success? If the answer to that question isn’t being tracked, then it’s certainly worth tracking. As you get more familiar with your product, you’ll see that certain behaviors are more important and you’ll start identifying those over time.
If you look outside of the gaming space at other tech companies, they have this notion of north star metrics. Facebook emphasizes DAU. Airbnb optimizes on nights booked. WhatsApp looks at the number of messages a user sends per day. There are a lot of differences in the metrics they track.”
Varying schools of thought
“There a few schools of thought on KPIs. One school is focused on the standard set like ARPDAU, retention, ARPPU. There’s a second school that looks at just about everything. There’s a third school that has a dehydrated set of standard KPIs mixed in with a set of custom KPIs on a per-game basis. But I’m increasingly hearing about a fourth school, which only use a few custom KPIs that are game dependent. This probably originated from Kabam. In this scenario, DAU or ARPDAU might not even be on the main dashboard. Instead, it may be regulars DAU or DAC.
“It’s worth pointing out that the school of thought you’re using depends on your organization’s maturity. If you’re a mature organization, you can look at custom metrics in a vacuum. But if you’re just starting out, leaning on game-agnostic standardized metrics will let you compare your game to your most successful title or other games out there.
Putting too many metrics front and center can make it hard to build strong data habits with your team and be distracting. Game KPIs are diagnostic, sure, but they’re also a jumping-off point. If a metric doesn’t truly make or break your game, you shouldn’t be shy about putting it somewhere people can look at it situationally.”
What to do when there’s an issue in KPIs
“When there’s an issue, first verify that the data can be trusted and that there indeed is an issue. All too often, game teams assume that the data can be trusted and they go down this rabbit hole of investigating, only to figure out that there was a data issue. Once the data has been verified, this is where root cause analysis can come into play, in order to really diagnose what the issue is.
Let’s say, you see a drop in revenue - something product managers are super familiar with. The goal here is to reduce revenue down into its core components and see how those core components contribute to that drop. You can reduce revenue into DAU and ARPDAU. But there are ways to reduce those metrics even further. Let’s say in this hypothetical, you see that the majority of the decrease is in the ARPDAU. From there, you can reduce ARPDAU into payer conversion rate and ARPPU. If you see the majority of the decrease is in ARPPU, you can reduce that further down into payments per payer and average transaction size. If payments per payers are up by 20% and average transaction is down by 40%, now you have something that’s a little more actionable that you can start investigating things that happened in the game in the last day or so that may be contributing to those fluctuations. It’s possible in this hypothetical, that a targeted offer you set up drove a decrease in average price and also drove an increase in payments, but not enough to offset the decrease in price.
Still, this doesn’t really tell you how to solve the issue. That’s where product manager intuition and roadmap prioritization comes into play - really understanding the live ops you’ve been running and how you can improve that going forward.
You tend to get to the heart of these problems by cleaving deeper and deeper into the thing that looks wrong. You’ll split things out by platform, and if the problem is evenly distributed across platforms, you understand it’s not a platform thing. Making those cuts until eventually, you’ll find one thing that’s dropped to zero while everything else is fine.”
Cross company communication is key
“The more context and understanding you have for how the game works, the better you’re going to fare. That’s why it’s important to communicate with your team - you have all these people who can help you understand, instead of causing a panic or launching separate investigations. Teams like QA, customer support, research, social, community all have insights into what’s going on in the game. Effective organizations are going to find a way to give them a voice.”
Is real-time data necessary?
“Real-time data comes at a significant cost from a performance perspective. That being said, there’s still a use case for real-time data, which is to react quickly to changes in your game. For example, at Scopely, we had a real-time monitoring system that would alert the product team through text of any notable changes in monetization that needed to be addressed right away. Here, the added cost of real-time data definitely paid for itself.”
Building data infrastructure
“You need to record the things that happen in your game, discard anything that’s invalid, aggregate all your valid data in a way that’s meaningful to a person, and actually put the data somewhere where it can be viewed. That means you’ll need a data pipeline that has components for filtering, processing, and storing the data as well as a reporting mechanism.”
Safeguards against bad data
“When your data comes through improperly, it’ll bring your work to a standstill. There are a few hugely impactful things you can do to improve data quality. One thing, which is less common than I’d have guessed is to get your QA team involved in analytics QA, so you can have as many eyes on the data as possible. We have lots of people looking at the player facing part of the game, but the analytics side is just as easy to grasp. You can write test cases and give the QA team guidance on areas to focus on with each update. You can also make sure that the expected tracking for your game has been formally defined down to the data type, so you don’t have calculation errors further down.
As your team grows, automating data processing will ensure everyone gets the same answer, instead of calculating it their own way. This will also reduce processing costs, maintenance, and troubleshooting, and improve the performance of all your reports and dashboards.
Data for scale and multiple titles
“Building out infrastructure can be tedious and slow, but it pays off if you can get repeat use out of it over time. You can accomplish this by standardizing your inputs and building a system that’s flexible to handle as many of those inevitable exceptions to your rules. You just need to make sure the spirit of the tracking is the same across titles. For example, you could make a key in your tracking called ‘combos created’ - in a puzzle game, you could use it to count the number of power gems you created on the gem board, and in a fighting game, you could use it to count the actual combo moves that are used in either case. This tracking can be used as an indicator of player skill, and make it easier to roll your work from one game to the next, even if they’re fairly different titles.”
Common mistakes for analysis
“The most common mistake in analysis is around priorities - or prioritizing the wrong type of analyses given the current state of the product. The stark reality is many game teams are resource-constrained, so you have to be methodical about the types of questions you want to prioritize answering. A good rule of thumb in determining analytics priorities is to think about the important decisions and data points you’ll need to make a sound decision.
Common mistakes for data infrastructure
“Not spending enough time thinking about data infrastructure early on is a big mistake. Data tends to be an afterthought in the production process, which leads to errors that require massive cleanup efforts. A lot of this can be improved by thinking about how you’re going to involve analysts in the game and what your expectations for them are.
Getting data analytics more involved in the entire development process is often not thought about. When you’re developing features, you need to think about what you want to track. Getting the perspective of an analytics person is hugely helpful.”
The events spec
“Data from your game undergoes a really long journey before it lands in a dashboard or report. If a single step fails, the whole process is in jeopardy. Data predictability is a key element to get this data journey to be a success. To achieve this predictability, your team can define what your tracking is going to look like. It works in the same way a car factory might have a spec - there’s certain tolerances your product has to have to be allowed out of the facility and on the store shelves. Having these definitions established gives your developers specific guidance on how to instrument the game, lets your QA team determine whether tracking is working as expected, and keeps wild data from contaminating your pipeline. By discarding events that don’t match what you’ve specified, it keeps everything downstream from getting contaminated.”
Want to learn what goes into making the world’s most popular games? Subscribe to LevelUp to never miss an update.
