How Telstra is applying machine learning to marketing mix modelling
- 26 October, 2017 15:00
Understanding the impact and optimal combination of marketing and media tactics for success is an increasingly complicated task for marketing functions. It’s even more difficult if you want to interpret the data in a quick enough timeframe to optimise, pivot or swap out marketing programs while they’re still in-market.
But this is exactly what Telstra is becoming able to do thanks to a new marketing mix modelling approach based on machine learning, rolled out in June.
Telstra director of research, insights and analytics, Liz Moore, told CMO the ASX-listed telco built an econometric approach to marketing mix modelling about five years ago based on the need to better understand the ROI of marketing at a strategic level. While it satisfied one organisational need – namely, giving the CFO a view of marketing ROI – it was an expensive exercise, taking 4-5 months to gather more than 300 data sources, plus months to build the modelling.
“Eight to nine months later, we could tell how effective your marketing mix was 12 months ago,” she said. “It wasn’t providing marketers with real-time flexibility of understanding what is going on in the marketplace. The challenge we wanted to solve, was that for every dollar we are spending, how effective is it? And how do we do that fast enough to take decisions off the insights being made around the CMO function around how to go to market?
“We knew continuing to take the traditional econometric approach wasn’t right. We were spending more than returns we were getting, given it was being delivered so late after the fact.”
Telstra went to market for a new approach that was fast, cost efficient and incredibly insightful, Moore said. In addition, any solution needed to sit in the ecosystem of tools being used measure marketing and not be done in a vacuum chamber. These include Aprimo’s Marketing Studio operations management tool, a bottom-up attribution modelling solution fuelling Telstra’s programmatic trading desk, brand and ad tracking, and also uplift over control measurements across every one-to-one marketing campaign undertaken by the team.
“That’s important because if we look at a campaign or channel has performed well or poorly, we know it can be for three reasons: There’s an investment issue; there’s a creative issue; or there’s an offer or competitive issue,” Moore commented.
The solution: Machine learning
Telstra partnered with Accenture to tap new machine learning technologies that could drive improved speed to insight. Known internally as ‘triple M’, the marketing mix modelling solution is refresh monthly, instead of every six months, and available on the desktop of users across the business within a month of closing off a month, once data sets are all in. The tool is tapping more than 4000 data variables, most of which are automated, bringing speed and agility that’s unprecedented, Moore said.
“It’s put in the hands of media buyers, marketers, CMOs and finance and to the board a close to real-time assessment of how marketing is going so we can make strategic decisions about whether we are investing in the right places, with that strategic overlay as well,” she said.
As part of the development process, Moore and her team worked with marketers, media buyers and agency partners to understand constraints and criteria, and devise modelling simulations and recommendations that both made sense and would be useful. It was also vital the tool provide a through-the-line, cross-segment, integrated view of marketing, Moore said.
Many of the data assets fuelling the model were already in the hands of the Telstra analytics team, including feedback from media buyers at a granular level. The problem was integration was largely manual, with some sent in spreadsheets and sent at different times. A key step in development of ‘triple M’ was working with providers to get data provided in a systematic, automated fashion, Moore said.
The modelling also includes Telstra data variables around sales, footfall, activations, deactivations, along with competitive landscape information.
“A key lesson is that many assume data feeds are going to be won and done in projects like this. But data is a bit like a garden, you have to constantly tender it,” Moore said.
Accenture Analytics managing director, Marek Rucinski, said it was vital to gain an understanding of the interaction of channels in terms of conversions and ROI.
“It’s a broader problem for marketers – how digital interacts with traditional channels,” he said. “Many are struggling with that intellectually. We now have a detailed picture of these cross-channel interactions, which can then be presented visually to the team.”
Rucinski said the tool incorporates three core data packets: Performance data; marketing spend data such as campaign details, native to advertising, reach; and controlled data sets, such as competitor performance, pricing movement. Driving the model is machine learning and open source platforms that allow data and models to be processed more quickly.
“It now takes us 11 days to run more than 900 models simultaneously. You can’t underestimate the processing power needed,” he explained. “It’s self-learning, and every cycle changes based on patterns. That introduces its own challenges as you have to ensure the team can explain what the model is generating.
“Other continual change is the business doesn’t stand still. This needs to be made to evolve with the business, otherwise you have the tail wagging the dog.”
As a case in point, Moore said Telstra’s launch of brand 3.0 last year half-way through development of the new marketing mix modelling tool changed the structure of how the business goes to market and meant a rebuild of all the models previously created.
“As the business changes and evolves, we have to continue changing the way we measure and think about marketing modelling,” she said. “Machine learning allows us to make those changes in a much more agile way.
Rucinski also described the presentation layer of Telstra’s new marketing mix modelling platform as “radically different”.
“We’ve moved from static reports and heavy desktop tools to something that is light on and fact-based, with different screens for different users such as the executives, who get a broader view; or a campaign manager, who can drill down into specific campaign and how it’s performed,” he said.
“The key to success is creating a sense of belief about the results. If it doesn’t relate to a marketer’s function or reality, they will ignore them. Telstra and the team worked to create a reference set that creates a common language. That’s a major achievement.”
While the maths behind the modelling is complex, Moore agreed it’s the cultural component that often proves the hardest part of rolling out such a solution.
“Without culture, the stickiness just isn’t there and you never get to your destination,” she said. “What was important was our CMO was 120 per cent behind this, wanting us to move to making decisions that are data and fact-based. It’s that support from the top that drove take-up throughout the organisation.
“It’s embedded in the way we look and assess campaigns now. People are now saying ‘have we have run it through triple M, what are the insights?’. Plus, it’s been built into the operating rhythm and a tool that supports the CMO.”
As part of the rollout, Moore’s team worked with Deakin University’s professor of marketing, Chris Dubelaar, to review maps.
“We were pushing the envelope so we wanted an academic who could test the thinking and provide input and work with us,” Moore said. “It was useful for the modelling and our team to have that because what we’re doing is so bleeding edge.”
Already, ‘triple M’ is bringing a degree of experimentation to Telstra’s above-the-line investment, Moore continued.
“We look at a channel giving us a higher ROI than thought, and this allows us to say let’s test that theory, and put more investment in one geography and see how it performs against another geography,” she said. “It’s a measurement tool for assessing experiments.”
But the tool itself isn’t the whole solution and both Moore and Rucinski emphasised how important it is to consider the wider marketing context.
“We also need to include qualitative and quantitative research data to see what it is telling us,” Moore said. “That’s the art beside the science. This is a bedrock and important part of our measurement ecosystem, but we need the other pieces to work well together too.”
To ensure its hundreds of marketers don’t “run down data rabbit holes”, Moore said the group has limited access to those who need to use this on a day-to-day basis. That includes the marketing media buyers and CMO leadership team. To help, the analytics team has conducted monthly deep dives, bringing in insights to support particular campaign planning and trading cycles.
There are also specific individuals in the operations team of each CMO function and the media agency utilising the modelling tool today.
Moore said the next big step forward will be moving the tool from being monthly to weekly.
“That allows us to pick up underlying performance view and assess on a weekly basis to allow for inflight tuning,” she said, adding the next phase is due in the New Year.