How to run marketability tests for your prototypes

Learn how to put your new game concept to the test – literally. In this class, you’ll understand if your game is marketable or not and whether you should continue building out the game or move onto the next idea.

Hi everyone, I’m Adam from ironSource and today I’m going to be talking about marketability testing your hyper-casual prototype.

I’ll be sharing insights straight from ironSource’s publishing solution, Supersonic, which in two years has published 35 hyper-casual mobile games. 25 of which reached the top 10 in the gaming charts in the US, so it’s fair to say that we know what we’re talking about!

In this four-part course, we’ll take you through the entire process of developing a hit hyper-casual game – from coming up with a concept all the way to launching it globally.

The last session was all about the different approaches for coming up with a new game concept. In this class, I’m gonna show you the next step – how to put this new concept to test, literally.

Knowing when to move on from research to actually choosing a concept and starting to prototype can be hard.

The truth is, you’ll rarely be 100 percent sure, but don’t become a victim of analysis paralysis. Once you’re reasonably confident in an idea, take the leap and start prototyping. You shouldn’t be investing a huge amount of time and resources into your prototypes…

During the prototyping phase, your goal is to get a sense of how well your gameplay resonates with users and to confirm the concept has potential. Marketability testing is the way you’ll do this.

Marketability shows the size of the potential audience for your game. Also, it predicts how well they’ll convert from an ad impression to an install on different UA channels. Remember the goal for hyper-casual games is to reach a very wide audience at a low cost.

Keep watching to learn your KPIs, and how to run a marketability test and analyze the results to see if your concept is a product-market fit that can scale.

KPIs to track

The first thing to know before you start marketability testing is which metrics to measure. The three most important are CPI, APPU, and retention. Let’s dive into each so you’ll know what they tell you and why it’s important.

CPI

CPI, or cost per install, is how much you’re paying for each install. CPI reflects the marketability power of your concept and is the most important KPI when prototype testing.

A low CPI is confirmation that many users understand and enjoy your game – and that your game can scale without breaking the bank. Aim for a CPI under $0.30 …

With a CPI in this range, you can start making game improvements that improve in-game metrics like retention and playtime instead of focusing only on lowering CPI.

On an SDK network, you should also monitor the IPM of your prototypes – the higher your game’s IPM, the stronger its marketability power. Analyze both the average IPM from the prototype test and the amount of app sources that generated an IPM above the average.

APPU

APPU, which stands for average playtime per user, is a metric created here at Supersonic as a more accurate prediction of a game’s LTV.

APPU, which is measured in seconds, tells you how much time a user spends on average in your game over the course of a lifetime, or over a specific time period like the first 7 days since install – way before you even start monetizing

It’s the closest signal of LTV because the longer that users play your game, the more ads they’ll watch, leading to more revenue.

To calculate APPU, we use the following formula: Playtime D0+ Retention D1XPlaytime D1*+ D2*.

Let’s put this to an example – check out the chart:

Here, APPU D7 shows the total ad impressions – if your game has a low APPU D7, then total impressions are low and LTV is likely to be low, too.

In this example, APPU D7 is above 2,600 seconds, indicating that users are playing the game for a long time and will view many ads.

The high APPU D7 here likely means a high LTV too. In general, you should aim for an APPU D7 of over 2,200 seconds – this threshold of APPU D7 indicates the game has enough monetization power to go to soft launch.

Retention

The third KPI you need to focus on in your prototype testing phase. You always want users to return to your game and spend more time playing – this means more opportunities for ad monetization.

Retention – specifically late retention – plays a big part in increasing your LTV per user. So make sure your prototype shows strong retention – Generally, you should aim for a D1 retention rate of 38% or higher on Android.

Draw the Line, for example, earned significant profit and reached the top 5 on iOS in the US.

When the game was first tested, it had an impressive D1 retention rate of 51%. Even though the CPI was high at $0.38, the high retention rate told us that users loved playing the game enough to keep coming back.

Building your creatives

Now you’re probably thinking about what you should actually be showing in your video ads. I’m about to tell you 5 best practices to help you do it right…and for deeper insights about nailing your videos at this stage, check out our blog post on how to build and optimize creatives for marketability testing.

Try out 4 different variations

We suggest creating 4 versions of your creative to test in the marketability stage. This way you can try out different ideas, without having an overwhelming amount of variations. Focus on core gameplay in your creatives, with minimal editing and additions.

Some design elements to test in your 4 variations include:

-Frame composition and camera angle

-A tutorial-like feature to make it easy to understand how to play, like a hand pointing at a button

-The right color palette for your game, making sure you have a good contrast between the main objects like the characters, enemies, and obstacles.

-Character design and how you illustrate them (e.g. cleaner, simpler, and less realistic)

You can even test elements not in your game yet – if these features go down well, you can design them into the build when you start growing your game in the next stages.

Focus on the first 3 seconds

The first 3-5 seconds of a marketability creative are the most important – not only do you need to grab users’ attention but you also need to show what the game is all about.

So focus on making your creatives as clear and easy to understand in the beginning.

The fail situation could be a winner

70-80% of the most successful creatives we’ve launched show a “fail situation”, which is when the player in the video loses the level.

In at least one of your creative variations, show failed gameplay in the first 3-4 seconds and end the video on the message or image that appears when users fail a level.

Bazooka Boy’s best-performing creative was a video showing a level fail again and again. During their marketability test, the game had an amazing CPI of just 15 cents and later went on to achieve an LTV of over $1.

Take a risk

Be sure to make one of the four creative versions quirky or unusual. Think outside the box on this one – the creative should still connect to your theme and gameplay, but it should be more unique than the rest of the creative set.

Sometimes it takes the wild card to crack the marketability test. When designing the creatives for Chat Master, for example, we tried a version using a voice-over.

We used audio for Air Control, too, putting realistic sound effects over the video creative. In this video, we showed a person in real life drawing the game’s concept, which was an unexpected way to design game creatives.

Adapt the creative to Facebook

We recommend running your first tests on Facebook, which we’ll go over in more detail shortly. Before that, know that your creatives should be designed so they fit with Facebook’s UI…you want the ad to feel and look native to the platform.

A simple way to do this is changing the frame to white in your creatives, to blend in with Facebook’s feed.

These tips in action

We’ve just gone over our tips for improving your hyper-casual creatives, but sometimes seeing is believing. Let’s look at two examples of creatives that decreased CPI using a mix of the tips.

For Stacky Dash, we started with a version that featured a pale green environment and had a CPI of $0.41. We tested a few iterations like we suggest in tip #1, and analyzed the CPI before deciding to focus on changing the color and saturation of the background. We kept the same situation (a level fail, like tip #3) but changed the color to a bright pink so it stood out on social (tip #5) and engaged more users, which led to a significant decrease in CPI – to $0.28.

Chat Master is another useful example. The first version on the left had a CPI of 40 cents. It had a stylized chat format and a pointer on screen.

We then thought outside the box and tried something unusual, like in tip #4, by adding an image of a real girl playing the game and speaking as she types. We also finished the creative on a fail message (tip #3) and changed the format (tip #1) so it looked like an iOS chat. The resulting version with our optimizations lowered CPI to $0.30.

Start testing – and keep testing new concepts until you succeed

Okay, so we’ve got the KPIs down and understand what goes into building creatives at this stage. Now let’s move on to the actual testing phase and best practices for that.

During the prototyping phase, your goal is to get a sense of how well your gameplay resonates with users and to confirm the concept has potential. Because Facebook is easy to integrate, cost-effective, and let’s you learn quickly, it’s typically the best place to begin your testing, so we’ll start here.

TLDR – the 4 things you need to get right when doing CPI tests on Facebook are:

1) the number of videos you use for the campaign

2) setting the right budget,

3) targeting the right audience

4. Duration of campaigns

Let’s unpack these in a bit more detail.

Number of videos

Each creative set should contain 4-8 videos. You shouldn’t test more than 10 at a time because any more than that will spread the audience too thin – and if each creative reaches an entirely different user base, you won’t be able to get a sense of the true marketability of your game.

Between 4 and 8 videos is the perfect amount to attract a wide audience and give you an understanding of how marketable your game concept is.

Budget and targeting

Set your daily budget based on the operating system: usually we suggest $15 per ad on Android and $20 per ad on iOS.

These target budgets make sure you’re able to reach a large enough audience without overspending.

As for targeting, hyper-casual games are meant to be accessible – so you need a large user base to scale quickly and profitably. When setting up your test, keep the default placements and audience data suggested by Facebook and select a wide age range.

Duration

Once you’ve set up your CPI test, let it run for 6 days. The learning phase on Facebook takes 3-4 days before CPIs stabilize – 6 days is enough time to give your game a fair chance to prove its marketability without letting the test go too long or cost you too much. After letting the test run, it’s time to analyze your results.

How to understand your test results

Use the metrics that Facebook provides you about your audience and the performance of each ad creative to understand whether you should keep or kill your game.

In terms of your audience, you’re looking to reach an audience that tends to be loyal to hyper-casual games and can be reached by SDK networks.

If the majority of the users clicking on your Facebook ads are from a specific demographic, it could be difficult to achieve scale because you’d be relying on a limited user base. The broader and more equally balanced your audience is, the better.

Now what about CPI? If you get between 50 cents and 35 cents, the concept is worth iterating on to see if it can be optimized. If your tests on Facebook are seeing CPIs of 35 cents and lower, it’s a good benchmark for marketability and you can begin building out your game and improving in-game metrics like retention and playtime.. Something to bear in mind here is that the CPI benchmark does depend on the CPI and CPM trend on Facebook which is highly affected by seasonality and competition.

Also, it’s worth knowing that your creatives are automatically shown on Facebook’s different ad placements: the Facebook feed, Instagram, and the Facebook Audience Network.

The more balanced the performance across placements, the better – it indicates your game will scale well across both social platforms and SDK networks when you move onto the next stages of launch.

You’ll test many prototypes, and only a few will show potential in terms of low CPI and good in-game metrics, like retention and playtime.

Since you’re going to be testing so many prototypes, limit the time and resources you put into each prototype.

The more you test, the more likely it is that you’ll find a marketable concept. When you find a concept that hits the right CPI and IPM benchmarks, you can begin building it out and optimizing in-game metrics. and setting yourself up for success at soft launch.