intent.ly Live! Panel Session: Incrementality

On 27th September we hosted our very first intent.ly Live event, an insightful day of content and networking. The highlight was a panel discussion on incrementality - experts from The Body Shop, Currys, Adtraction and Gen3 shared their thoughts on this hot topic.

Katy Hughes, intent.ly's Client Success Director was joined by Natasha Phillips, Global Head of Affiliates and Partnerships at The Body Shop, Anthony Clements, Managing Partner of Adtraction, Bruce Clayton, Senior Vice President at Gen3 and David Evans, Senior Paid Marketing Manager at Currys.

How important is incrementality within your business and how regularly does it come up in conversation with your clients?

BRUCE: As agencies and clients we've been addressing incremental spend and conversions for about two years, but talk of incrementality around new customers has thrusted it into the spotlight. As an agency, we require more insight and analysis from clients on what the outcome of sales are  further down the line to investigate.

ANTHONY: From a network point of view, you tend to look at it as lots of different sized clients, and the vast majority will certainly ask themselves the question - whether they've just started in affiliate or partner marketing, or they've been in the industry for a long time - “how valuable is this in the context of other marketing channels?” 

Often we see clients ask that question in terms of attribution rather than incrementality. Maybe a textbook would say the two are two very separate things and should be treated separately, and I personally probably agree. But when a client tries to answer this, the two things always get conflated. Certainly in affiliate marketing that's a challenge we have to work through because you have the added aspect of so many different types of marketing in play under one channel banner. So that’s a really interesting challenge.


From a brand perspective, are you mostly measuring incrementality within affiliates or also across other channels equally? 

DAVID: For Currys it's the whole marketing mix, even ATL. Obviously they can't measure as well as we can but for us, in establishing where we're going to get the best return for our buck, it has to be included in the whole marketing landscape. It makes it a little bit more challenging just because their measurements are very different to what we can do, but it's essential. At the moment everyone's arguing that the next pound is best spent on their channel so we have to justify that, which is why incrementality is so important in helping to tell that story.


What do you deem to be incremental? Is it new customers, does it change week to week or by channel, or by KPI for example?

NATASHA: It starts with revenue, then new customers can be layered on top. It’s really trying to understand how the money that you're spending is returning and you can dig into that further afterwards. The main story is that if you can prove your channel is more incremental, then you'll get more budget.

DAVID: You have to continue to test, because as soon as you make an adjustment to how you’re operating, it's impacted again. That's what we did last year - we calculated an incremental number and adjusted our budget significantly, but now we've made those changes we can prove the marginal performances at a certain level. But we're then trying to measure how that’s changed because the landscape’s also changed. We're quite unique in that we' also look at impacts in stores as well, which is interesting if you can link up the omnichannel piece.

NATASHA: I think this also ties into attribution because as you make a change, how you're attributing the sales also changes. So as you say, you have to continuously be testing and monitoring. Incrementality and attribution will never be separate as they impact each other. 


Do you run incrementality tests in silo or do you combine affiliate results with other channels? 

NATASHA: Within each channel, we'll be doing incrementality tests. So with the activity that we're running with intent.ly, your team will be looking at that in detail for us, but then we also do incrementality tests across two or three other channels as well. So we have all of these different tests going on at the same time. 

DAVID: I think we're at the beginning of the incrementality journey and learning all the time. What's made it very tricky over the last year is that trends have changed significantly - like the cost of living crisis, but we've also seen stores doing better than expected, so that makes forecasting and planning really challenging. Seasonality and weather can also impact all those factors as well. 

From an agency or network perspective, have marketing strategies changed based on incrementality test results?

BRUCE: We're an agency with a lot of clients and we work as closely as possible with them to make sure the channels are doing what they want them to. But we're a little bit beholden to the information we get back from them - obviously we have a set of levers we can pull to try and influence certain outcomes. But again, we're only as good as the data we get back and it needs to be run for a period of time - at least one validation cycle - so we can say "here is a batch of affiliate orders, can you give us the outcomes on these"? 

We can measure things like conversion by partner or partner type, spend etc in a fairly quick and easy feedback loop, but the wider incrementality piece requires us to work with partners like intent.ly. We can do a certain amount of it by having access to GA but again, a lot of what you get out is only as good as what you actually put in. 

We're starting to have more involved conversations around planning for Q4 and 2024 than we ever have before. So hopefully incrementality is something we can be more involved in from the start, and we can help inform the process.

ANTHONY: Partners like intent.ly benefit from being able to run good incrementality tests and incremental lift tests on activity in a way that most other participants in the channel just can’t do to the same level, like cashback sites. So from a strategy point of view, the partnerships where good channel silo testing can be done will benefit greatly from good campaigns that deliver value.

Is there a standard metric to measure incrementality across channels in terms of the control and test group splits, in the quest for one single source of  truth? 

DAVID: I think it varies channel by channel and based on the data you have available. So for example for PPC it's a lot easier to do a dark trial where you just turn it off completely and it's a lot easier to measure the impact. You’re dependent on tech partners as well, and their testing capabilities. Unfortunately, there's no easy answer and there's no one-size-fits-all approach. If you haven't got the right partner or the right data source to help you do the testing correctly, then it's going to be questioned constantly. 

NATASHA: Ideally you have a small five percent test group, ongoing, updating and improving over time. But sometimes you don't have enough data, so you need to broaden that out. For other channels it can be easier, such as PPC for example, to constantly test the copy. It would of course be great to be able to do that with everything, but it takes a lot more work.


What length of data capture period do you recommend for incrementality testing? Do you look at the entire shopper journey or drill down into key KPIs only?

NATASHA: To get really clear results, it has to be really simple. You may see benefits across other metrics, but you have to be measuring on one thing because if not you can get really caught up in other details, like the AOV or conversion rate increase etc. But actually you weren't testing either of those originally. It's best to have a key KPI that you want to see an improvement in - the rest is still a great story but the tests need to be ongoing and consistent.  

The other thing is we don't always know if the website or PPC teams are changing something else that could affect it. So you do need to look at everything over time, and how the conversion rates have been impacted.

How much responsibility do you think the networks have in helping to prove incrementality for their clients?

ANTHONY: I think it's the responsibility of individual companies who are spending money in the affiliate channel to understand how it works for them. The role of the network is to then make sure it provides the right kind of data and the right kind of marketing execution to help them do that. Ultimately, if the marketing is good then we should believe that it's going to deliver good incremental value to the client.

I think publishers also need to be more open to doing the kind of incremental value testing that advertisers require to be able to demonstrate the value of that particular partnership. This will vary depending upon the partnership, just like the metrics that you choose to use to determine success are going to be very different depending upon the partnership. With one partner model, it might be about volume or new customers; with a partner like intent.ly you might be testing around AOV or conversion rate, depending upon the type of campaigns.

I do think it is quite incumbent on the networks to be able to help the advertiser by saying ‘this is the kind of marketing you should be doing because we know from empirical testing that it delivers incremental value’. I don't think we should rely on Google Analytics to tell us that answer because it's certainly not going to.

BRUCE: I think from an agency perspective, it's a mosaic approach working with the client at the moment to determine what it is they want to achieve, then finding a way to measure that. A lot of our brands are smaller so they’re looking for external solutions. Certainly the more information I can get as an agency and the more insight I have to inform a strategy and take actions on their behalf, the better. A lot of it comes down to what can reasonably be captured in affiliate tracking - another moving target.

I think the network should be a major part of this moving forward - and it's as much about functionality as it is about support. Hopefully the work they do with larger brands will mean the knowledge trickles down to the sort of things that agencies and smaller clients would find really, really useful in answering these challenges.

NATASHA: I think what would be useful is having examples, case studies or templates to show the ways that you can test it. So with a certain type of partner, this is the type of thing you could do. 

"The joy we have working with partners like intent.ly is being able to develop incrementality testing with you, because different departments and category teams at Currys require different levels of data. So it's us working in partnership to create a model that works not just for you, but also for our relevant teams and what they require."

With Black Friday and Cyber Monday just around the corner, now is the time to start planning your campaigns. intent.ly helps you to analyse shopper insights, and offer solutions and strategies to target them at all stages of the purchase funnel. Book a call to learn more or get in touch with your account manager today.