Big Data is a solution looking for a problem: Gartner

Data is not always an asset, and Big Data is not always the answer

Big Data is forecast to drive US$34 billion of IT spending in 2013 and create 4.4 million IT jobs by 2015, but it is currently still a solution looking for a problem, according to analyst firm, Gartner.

While businesses are keen to start mining their data stores for useful insights, and many are already experimenting with technologies like Hadoop, the biggest challenge is working out what question you are trying to answer, said Debra Logan, VP Distinguished Analyst at Gartner.

"Even banks are not doing Big Data in a production sense, because most of what they've got is pretty well organised and in mainframes," she said.

"Certainly they're investigating it, they're wondering what it means, they want to learn about that kind of technical architecture and the kinds of methods you use to programme and do analytics on it, but it's still in the early phases."

Indeed, both Deutsche Bank and HMRC recently admitted that they are struggling to find a way to unravel data from legacy systems to allow integration with newly created big data systems based on Hadoop technology.

Logan said that the most advanced industry sector in terms of Big Data is retail, because of the things it does with radio-frequency identification (RFID), supply chain management and loyalty cards. These, according to Logan, are "real Big Data problems".

A lot of other industries are simply looking for ways to manage and monetise their data assets - and Big Data is not always the answer, according to Logan. There are plenty of off-the-shelf software programs that can visualise large data sets, and in some cases the best solution may be to simply throw some of the data away.

"Not all data is an asset, and if it's not an asset then it's just costing you money. The first thing that has to happen is you have to make decisions about what could not possibly be useful," said Logan.

"Again, it depends what question you're asking. What is it that a massive set of data tells you about a particular problem that a more reasonable set of data about a particular problem doesn't tell you?"

Of course, in regulated industries it is not always possible to throw away data, so a decision then needs to be made on whether to start probing the data to see if it can yield more value. This is where business and technology need to start working much closer together, in order to understand the issues.

Organisations are now crying out for employees that not only have the IT skills to do the data processing, but also the analytical mind and the business nous to work out how to extract insights. Known as 'data scientists' or 'chief data officers', people with these combined skills are few and far between.

"A couple of clients basically had to learn it all themselves and do it all themselves, because people who do this stuff are currently working at CERN," said Logan.

"These people have mathematical and statistical modelling capabilities - that's not how we're training developers. In this day and age people barely know how to program, let alone do the mathematical modelling that they have to do."

She added that there is a huge gap in the market, but organisations should be wary of anyone waving a card around, claiming to be a data scientist. "You really have to dig into that and see if they actually know what they're talking about," she said.

An alternative is to take team approach, where the team includes a programmer, a lawyer and somebody who understands the business. By giving these people access to the data, and letting them play around with it, there is a chance that they may make some interesting discoveries.

"Businesses understand that technology is the answer, but what is the question? Let's just get these guys to experiment and see what they come up with, which is the way that innovation happens," she said.

There is currently no right or wrong way to 'do' Big Data, and organisations in different industries need to forge their own path. However, before plunging in head-first and investing lots of money in technologies like Hadoop, organisations might benefit from working out what they are trying to achieve.

Follow CMO on Twitter: @CMOAustralia.

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Show Comments

Blog Posts

Building a human-curated brand

If the FANG (Facebook, Amazon, Netflix, Google) sector and their measured worth are the final argument for the successful 21st Century model, then they are beyond reproach. Fine-tuning masses of algorithms to reduce human touchpoints and deliver wild returns to investors—all with workforces infinitesimally small compared to the giants of the 20th Century—has been proven out.

Will Smith

Co-founder and head of new markets, The Plum Guide

Sustainability trends brands can expect in 2020

​Marketers have made strides this year in sustainability with the number of brands rallying behind the Not Business As Usual alliance for action against climate change being a sign of the times. While sustainability efforts have gained momentum this year, 2020 is shaping up to be the year brands are really held accountable for their work in this area.

Ben King

CSR manager & sustainability expert, Finder

The trouble with Scotty from Marketing

As a Marketer, the ‘Scotty from Marketing’ meme troubles me.

Natalie Robinson

Director of marketing and communications, Melbourne Polytechnic

It's a pretty interesting article to read. I will learn more about this company later.

Dan Bullock

40 staff and 1000 contracts affected as foodora closes its Australian operations

Read more

If you think it can benefit both consumer and seller then it would be great

Simon Bird

Why Ford is counting on the Internet of Things to drive customer engagement

Read more

It's a good idea. Customers really should control their data. Now I understand why it's important.

Elvin Huntsberry

Salesforce CMO: Modern marketers have an obligation to give customers control of their data

Read more

Instagram changes algorithms every time you get used to them. It really pisses me off. What else pisses me off? The fact that Instagram d...

Nickwood

Instagram loses the like in Australia; industry reacts positively

Read more

I tried www.analisa.io to see my Instagram Insight

Dina Rahmawati

7 marketing technology predictions for 2016

Read more

Latest Podcast

More podcasts

Sign in