How The Smith Family built a culture of marketing experimentation

Head of marketing shares how experimentation in campaigns and journeys is helping to deliver greater impact

Giving staff the space to experiment, the systems and KPIs to support a test-and-learn culture and an agile structure oriented around outcomes has proven a winning formula for The Smith Family’s marketing team.

Speaking at this week’s ADMA Global Forum, The Smith Family head of marketing, Lisa Allan, and agency partner, Douglas Nichol of The Works, took to the stage to talk through how marketers can build a culture of experimentation and its importance to the modern digital marketing approach.

For The Smith Family, the triumvirate of space, systems and structure drive marketing experimentation, Allan said. The not-for-profit has been committed to experimentation over the past four years as it works to support more than 180,000 children in poverty each year.

“You need to give team members space and the opportunity to think about how to do things differently. What marketing we can do different, what might that activity look like, or the next thing on from what we did last time,” she said. “Regularly, we’re also looking for team members when they join to have a curious nature to ensure they are willing to look at continual optimisation, not just execution.”

Systems is about supporting performance management systems, and Allan noted the not-for-profit has a test-and-learn KPI for teams as well as individuals team members. It also uses Visio Boards and hypothesis cards to build a knowledge base. This creates reference points, allowing staff across the organisation to see current tests as well as what’s been achieved in the past.

“We also talk about ‘flearning’, or failed learning, and bring that into our team meetings. We’re able to present back on things that didn’t quite go to plan and what we learnt from that,” Allan continued. “Sometimes there’s a lot more richness there than in things that did go well.”  

In terms of structure, Allan is running two agile squads in marketing, one focused on acquisition and the other on retention and loyalty. These cross-functional units look to iterate in two-weekly sprints in order to deliver more value to customers in the form of minimum viable products.  

Guidance principles

Both Nichol and Allan shared six overarching principles they’d also devised as integral to the marketing experimentation approach. The first, Nichol said, is understanding the right frequency of testing to foster a test-and-learn culture. He pointed to Harvard Business Review data, which showed 12.6 per cent of US digital marketers running ongoing controlled experimentation. Those running 15 experiments a year enjoyed a 45 per cent increase in campaign performance.

For enterprise-level clients, running at least 20 experiments per month is critical to building a knowledge base that more accurately informs and predicts the outcomes of future tests.

“So when you are dreaming up the next test, you can run the prediction model and get a really good idea on if it’s an intelligent test to run in market,” Nicol said.

The second principle both speakers advocated was to sweat your testing hypothesis. Allan said the hypothesis needs to be well considered, easy to measure and specific, and highlighted The Smith Family’s ‘test cards’ as a key ingredient here. These follow a hypothesis approach and summary: We believe (the target market) will do (this action) for/because (this reason). Testing is about verifying that hypothesis summary through the specific action and key metrics that show whether the hypothesis improves results against current metrics, Allan explained.

A third principle is that over testing in any one campaign can lead to data bias, while the fourth was the trade-off between execution of experimentation between knowledge and speed. Allan flagged statistical significance calculators as a useful tool for finding this balance.

“What you want to make sure of is you are not doing something that takes too long, when you could iterate and do smaller tests to build learnings and get what you are aspiring to,” she said.

Another piece of advice is to avoid being myopic in your view of testing. “Testing isn’t everything… you need to understand how testing fits in the wider measurement landscape,” Nicol said. He pointed to marketing experiments running through the funnel and alongside measures such as attribution and media mix modelling, an engagement index and brand tracking.

And for Allan and Nicol, it’s important not to sweat the small stuff. “Start with biggest hypothesis, then optimise as you go down the list,” she said.

Four success factors were outlined by the duo when conducting marketing experiments. Audience was top of the list at 40 per cent, with Allan stressing finding and targeting the right audiences efficiently with optimal channel mix as the first port of all for experimentation.

Ensuring your creative is grounded in target market insights and the needs, wants and desires of your customers is also critical to success. “It’s important to think about a proposition that’s easy to understand and that’s communicated well. And remove all barriers to response,” she said.  

Delivery, or the right frequency across channels to get engagement, is another must. “You can undercook and not get conversion, or you can overcook frequency and see diminishing returns,” Nicol said.

“Also, understand the role of brand in uplifting response rates. In our experience, you can get a lift of up to 18 per cent by running brand work at same time as your digital engagement work.”

It’s only after considering these factors that marketers should think experiment with variables such as colour, placement of buttons, words and copy tone. “They do have validity but best to deal with as you finesse your options,” Allan said.

At The Smith Family, this culminates in a flywheel which goes from hypothesis creation to the test use case objective setting, designing and building the test, results analysis, and the learning card detailing what has been observed, learned to inform next actions.

Allan shared several examples of testing in practice with The Smith Family campaigns. One notable journey example was a sponsorship product test related to sponsors sending birthday or Christmas wishes to their sponsored child. This was previously done via physically mailed packs with cards. As a result of the pandemic, The Smith Family converted this to online, using email and personalised Web pages with pre-populated forms and SMS reminders. The result was a 24 per cent lift in response rate and 60 per cent increase overall for that pack.

As dollars were going to students directly, this didn’t translate to a specific commercial outcome for The Smith Family. But as Allan made clear, it provided significant connection between sponsor and child.

“And I know those sponsors are three times more likely to stay if the sponsor and child are writing to each other,” she said. “So it’s important to look across multiple factors.”  

Through all of this, Allan also said The Smith Family had to be realistic about focusing on experiments that supplied biggest impact.

“It’s about looking at ROI and being smart with dollars and investing behind the stuff in what we think will drive the outcome,” she added.  

Credit: ADMA

Don’t miss out on the wealth of insight and content provided by CMO A/NZ and sign up to our weekly CMO Digest newsletters and information services here.  

You can also follow CMO on Twitter: @CMOAustralia, take part in the CMO conversation on LinkedIn: CMO ANZ, follow our regular updates via CMO Australia's Linkedin company page   


Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Show Comments

Latest Videos

More Videos

Nice blog!Blog is really informative , valuable.keep updating us with such amazing blogs.influencer agency in Melbourne

Rajat Kumar

Why flipping Status Quo Bias is the key to B2B marketing success

Read more

good this information are very helpful for millions of peoples customer loyalty Consultant is an important part of every business.

Tom Devid

Report: 4 ways to generate customer loyalty

Read more

Great post, thanks for sharing such a informative content.

CodeWare Limited

APAC software company brings on first VP of growth

Read more

This article highlights Gartner’s latest digital experience platforms report and how they are influencing content operations ecosystems. ...

vikram Roy

Gartner 2022 Digital Experience Platforms reveals leading vendor players

Read more

What about this one I think it's a great platform providing a lot of options, you can collect different data and work w...

Salvador Lopez

Gartner highlights four content marketing platform players as leaders

Read more

Blog Posts

Marketing overseas? 4 ways to make your message stick

Companies encounter a variety of challenges when it comes to marketing overseas. Marketing departments often don’t know much about the business and cultural context of the international audiences they are trying to reach. Sometimes they are also unsure about what kind of marketing they should be doing.

Cynthia Dearin

Author, business strategist, advisor

From unconscious to reflective: What level of data user are you?

Using data is a hot topic right now. Leaders are realising data can no longer just be the responsibility of dedicated analysts or staff with ‘data’ in their title or role description.

Dr Selena Fisk

Data expert, author

Whose responsibility is it to set the ground rules for agency collaboration?

It’s not that your agencies don’t have your best interests at heart – most of them do. But the only way to ensure they’re 100 per cent focused on your business and not growing theirs by scope creep is by setting the guard rails for healthy agency collaboration.

Andrew Pascoe

Head of planning, Hatched

Sign in