How The Smith Family built a culture of marketing experimentation

Giving staff the space to experiment, the systems and KPIs to support a test-and-learn culture and an agile structure oriented around outcomes has proven a winning formula for The Smith Family’s marketing team.

Speaking at this week’s ADMA Global Forum, The Smith Family head of marketing, Lisa Allan, and agency partner, Douglas Nichol of The Works, took to the stage to talk through how marketers can build a culture of experimentation and its importance to the modern digital marketing approach.

For The Smith Family, the triumvirate of space, systems and structure drive marketing experimentation, Allan said. The not-for-profit has been committed to experimentation over the past four years as it works to support more than 180,000 children in poverty each year.

“You need to give team members space and the opportunity to think about how to do things differently. What marketing we can do different, what might that activity look like, or the next thing on from what we did last time,” she said. “Regularly, we’re also looking for team members when they join to have a curious nature to ensure they are willing to look at continual optimisation, not just execution.”

Systems is about supporting performance management systems, and Allan noted the not-for-profit has a test-and-learn KPI for teams as well as individuals team members. It also uses Visio Boards and hypothesis cards to build a knowledge base. This creates reference points, allowing staff across the organisation to see current tests as well as what’s been achieved in the past.

“We also talk about ‘flearning’, or failed learning, and bring that into our team meetings. We’re able to present back on things that didn’t quite go to plan and what we learnt from that,” Allan continued. “Sometimes there’s a lot more richness there than in things that did go well.”  

In terms of structure, Allan is running two agile squads in marketing, one focused on acquisition and the other on retention and loyalty. These cross-functional units look to iterate in two-weekly sprints in order to deliver more value to customers in the form of minimum viable products.  

Guidance principles

Both Nichol and Allan shared six overarching principles they’d also devised as integral to the marketing experimentation approach. The first, Nichol said, is understanding the right frequency of testing to foster a test-and-learn culture. He pointed to Harvard Business Review data, which showed 12.6 per cent of US digital marketers running ongoing controlled experimentation. Those running 15 experiments a year enjoyed a 45 per cent increase in campaign performance.

For enterprise-level clients, running at least 20 experiments per month is critical to building a knowledge base that more accurately informs and predicts the outcomes of future tests.

“So when you are dreaming up the next test, you can run the prediction model and get a really good idea on if it’s an intelligent test to run in market,” Nicol said.

The second principle both speakers advocated was to sweat your testing hypothesis. Allan said the hypothesis needs to be well considered, easy to measure and specific, and highlighted The Smith Family’s ‘test cards’ as a key ingredient here. These follow a hypothesis approach and summary: We believe (the target market) will do (this action) for/because (this reason). Testing is about verifying that hypothesis summary through the specific action and key metrics that show whether the hypothesis improves results against current metrics, Allan explained.

A third principle is that over testing in any one campaign can lead to data bias, while the fourth was the trade-off between execution of experimentation between knowledge and speed. Allan flagged statistical significance calculators as a useful tool for finding this balance.

“What you want to make sure of is you are not doing something that takes too long, when you could iterate and do smaller tests to build learnings and get what you are aspiring to,” she said.

Another piece of advice is to avoid being myopic in your view of testing. “Testing isn’t everything… you need to understand how testing fits in the wider measurement landscape,” Nicol said. He pointed to marketing experiments running through the funnel and alongside measures such as attribution and media mix modelling, an engagement index and brand tracking.

And for Allan and Nicol, it’s important not to sweat the small stuff. “Start with biggest hypothesis, then optimise as you go down the list,” she said.

Four success factors were outlined by the duo when conducting marketing experiments. Audience was top of the list at 40 per cent, with Allan stressing finding and targeting the right audiences efficiently with optimal channel mix as the first port of all for experimentation.

Ensuring your creative is grounded in target market insights and the needs, wants and desires of your customers is also critical to success. “It’s important to think about a proposition that’s easy to understand and that’s communicated well. And remove all barriers to response,” she said.  

Delivery, or the right frequency across channels to get engagement, is another must. “You can undercook and not get conversion, or you can overcook frequency and see diminishing returns,” Nicol said.

“Also, understand the role of brand in uplifting response rates. In our experience, you can get a lift of up to 18 per cent by running brand work at same time as your digital engagement work.”

It’s only after considering these factors that marketers should think experiment with variables such as colour, placement of buttons, words and copy tone. “They do have validity but best to deal with as you finesse your options,” Allan said.

At The Smith Family, this culminates in a flywheel which goes from hypothesis creation to the test use case objective setting, designing and building the test, results analysis, and the learning card detailing what has been observed, learned to inform next actions.

Allan shared several examples of testing in practice with The Smith Family campaigns. One notable journey example was a sponsorship product test related to sponsors sending birthday or Christmas wishes to their sponsored child. This was previously done via physically mailed packs with cards. As a result of the pandemic, The Smith Family converted this to online, using email and personalised Web pages with pre-populated forms and SMS reminders. The result was a 24 per cent lift in response rate and 60 per cent increase overall for that pack.

As dollars were going to students directly, this didn’t translate to a specific commercial outcome for The Smith Family. But as Allan made clear, it provided significant connection between sponsor and child.

“And I know those sponsors are three times more likely to stay if the sponsor and child are writing to each other,” she said. “So it’s important to look across multiple factors.”  

Through all of this, Allan also said The Smith Family had to be realistic about focusing on experiments that supplied biggest impact.

“It’s about looking at ROI and being smart with dollars and investing behind the stuff in what we think will drive the outcome,” she added.  

Credit: ADMA

Don’t miss out on the wealth of insight and content provided by CMO A/NZ and sign up to our weekly CMO Digest newsletters and information services here.  

You can also follow CMO on Twitter: @CMOAustralia, take part in the CMO conversation on LinkedIn: CMO ANZ, follow our regular updates via CMO Australia's Linkedin company page