Why you should be embracing a test-and-learn culture

The concept of failing fast and often is inherent in Agile software methodologies, but can it also be applied in marketing? We investigate why CMOs need to start testing

test and learn
test and learn

There is a mantra in the technology startup community that says failure is actually a good thing. That people who try something and fail are actually better investments than those who try nothing, or who have never known failure.

This ethos is embodied in the Agile methodology used by software development teams in countless market disruptors, from Amazon to Zappos. The Agile methodology places emphasis on short development cycles and constant testing – the learnings from which are fed back into the development cycle.

The most dramatic manifestation of this test-and-learn model can be found within Amazon, whose chief technology officer, Dr Werner Vogels, last year claimed his company could be testing up to 250 variables on its sites at any one time.

But it is a concept that is struggling to gain acceptance within the broader Australian marketing community. Where it has found favour tends to be within those organisations that have also adopted agile software development. And not surprisingly, these are primarily online-focused organisations such as SEEK, Wotif (recently acquired by Expedia) and REA Group.

REA Group: Making test-and-learn a philosophy

According to REA Group’s group manager for marketing strategy, Natalie Feehan, test-and-learn is used in everything from optimising digital advertising and electronic direct mail to tweaking its events and outdoor campaigns. She says test-and-learn is much more than just a process within the marketing department – it is a philosophy.

“Certainly here the culture is very much that we would rather people gave it a go and tried something different than just stuck to traditional methods,” she says.

With the company sending out emails numbering up to 100,000 each month, it first isolates small groups numbering in the hundreds to test different iterations of the communication. Variations might appear in fields such as the subject, title image, layout and content order. The response rates are then evaluated and used to tweak the eventual mass communication to its database.

here the culture is very much that we would rather people gave it a go and tried something different than just stuck to traditional method

REA Group's Natalie Feehan

“We are trying to leverage every send essentially, and get the highest engagement we can,” Feehan says. “In our email channel, it has been very successful in quickly accelerating engagement.

“In some cases we have doubled our open rates.”

While not all forms of communication provide such immediate results, the test-and-learn philosophy at REA Group means even TVCs and events are closely monitored to determine how minor changes compare to previous iterations. In these situations, traditional measures such as brand tracking are used, with REA Group pushing the frequency of polling to weekly.

Feehan says it should come as no surprise that REA Group also uses the Agile software development methodology.

“Any business working with an Agile methodology understands the value of iteration,” she says. “So philosophically they are aligned to thinking that things are fluid and changing. If that philosophy extends right through to marketing, people are generally not as afraid to fail, because there is openness and willingness and support from the organisation to continuously iterate.”

Getting up to speed

But while the test-and-learn mantra is deeply ingrained within REA Group, it has yet to work its way into the marketing departments of most Australian organisations.

Accordingly to Adobe’s APAC director of transformation and digital strategy, Mark Henley, the reason for this may be that many organisations are still struggling to get a grasp on the flood of metrics their online channels deliver, let alone using this data to effectively tweak campaigns in development.

“I think it is aspirational for most,” he comments. “What we are seeing is a shift in the last 12 to 18 months from people asking what is transformation and how does it affect them, to people being able to talk about the concepts of content personalisation, big data, optimisation and testing - but not knowing how to execute.

“So there is a growing consciousness of the necessity. The execution is something people are thinking about but haven’t worked through properly.”

Even more difficult is the execution of test-and-learn across online and offline channels.

“At the moment, there generally isn’t strong experience and capability in organisations to even do the simple stuff on a single channel,” Henley says. “Customer parsing is not an advanced science by any means.”

According to Deloitte Digital partner, Steve Hallam, the technology supporting test-and-learn has made huge leaps in the past five years, driving older clients to investigate it more seriously.

“A lot of large corporates are investing in it – particularly in the retail industry, where ecommerce conversion is most important,” he says. “But other incumbents in industry moving to digital-based models are also definitely investing in this technology.”

Related: 70 per cent of your marketing tests should fail, says head of Data@Ogilvy

It’s time for CMOs to embrace agile as a noun

Deloitte: The 5 disruptive trends of the digital age

Hallam says it is possible for brands to get a lot of traction - in the range of 20 per cent response improvements - with even a little data.

“You need to learn how to use this technology effectively and what parts of your market segment are going to be most effective,” he says. “And it is a transition. Amazon is very ahead of the game because the company has invested over a long time. Where companies are making the right level of investment, it is paying return.

“The investment you make in the digital technology to get the insight needs to be matched with the capacity and ability for the organisation to change.”

Up next: How the digital advertising community uses test-and-learn to drive uplift

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Show Comments

Latest Videos

More Videos

More Brand Posts

What are Chris Riddell's qualifications to talk about technology? What are the awards that Chris Riddell has won? I cannot seem to find ...

Tareq

Digital disruption isn’t disruption anymore: Why it’s time to refocus your business

Read more

Enterprisetalk

Mark

CMO's top 10 martech stories for the week - 9 June

Read more

Great e-commerce article!

Vadim Frost

CMO’s State of CX Leadership 2022 report finds the CX striving to align to business outcomes

Read more

Are you searching something related to Lottery and Lottery App then Agnito Technologies can be a help for you Agnito comes out as a true ...

jackson13

The Lottery Office CEO details journey into next-gen cross-channel campaign orchestration

Read more

Thorough testing and quality assurance are required for a bug-free Lottery Platform. I'm looking forward to dependability.

Ella Hall

The Lottery Office CEO details journey into next-gen cross-channel campaign orchestration

Read more

Blog Posts

Marketing prowess versus the enigma of the metaverse

Flash back to the classic film, Willy Wonka and the Chocolate Factory. Television-obsessed Mike insists on becoming the first person to be ‘sent by Wonkavision’, dematerialising on one end, pixel by pixel, and materialising in another space. His cinematic dreams are realised thanks to rash decisions as he is shrunken down to fit the digital universe, followed by a trip to the taffy puller to return to normal size.

Liz Miller

VP, Constellation Research

Why Excellent Leadership Begins with Vertical Growth

Why is it there is no shortage of leadership development materials, yet outstanding leadership is so rare? Despite having access to so many leadership principles, tools, systems and processes, why is it so hard to develop and improve as a leader?

Michael Bunting

Author, leadership expert

More than money talks in sports sponsorship

As a nation united by sport, brands are beginning to learn money alone won’t talk without aligned values and action. If recent events with major leagues and their players have shown us anything, it’s the next generation of athletes are standing by what they believe in – and they won’t let their values be superseded by money.

Simone Waugh

Managing Director, Publicis Queensland

Sign in