Big Data hype still here, but maturity beckons

IDC Analyst: Big data is about decision-making, not just technology for technology's sake

Few tech industry buzzwords have gotten as vigorous a workout as 'big data', but while the hype remains plentiful, it is starting to give way to real-life successes as well as formal ways companies can develop big data strategies, according to a number of IDC analysts.

"We're at this period now where there's certainly a lot of hype, a lot of promise," said IDC analyst, Dan Vesset, during a panel session at the research firm's Directions 2013 conference in Boston in March. "The question is what's reality, and what can and should companies do in the near and short term?"

IDC, which is owned by IDG News Service's parent company International Data Group, defines big data as information-driven, tactical decision-making resulting in new economic value, derived from large data volumes from a wide variety of sources, Vesset said.

The first point is crucial, according to Vesset.

"We can be installing all kinds of technology but the important thing is to improve the decision-making," Vesset said. "You can have the greatest Hadoop deployment in the world but it's not going to be enough," he added, referring to the popular open-source data-processing framework that is synonymous with the big data movement.

Progressive is one example of a company using big data projects to transform their business, said IDC analyst, Michael Versace, who presented with Vesset. Using detailed information about customers' driving habits, the insurer has created a usage-based model that defines a policy's price down to the individual, he said.

Progressive gets the data through a device a driver plugs into a car's diagnostic port, according to its website. It can track how often customers slam on their brakes, drive late at night and other possibly risky driving habits. If the data shows a customer is driving safely, they can get significant discounts on their insurance.

Meanwhile, a number of challenges stand in the way of companies wishing to launch successful big data projects of their own, according to Vesset. These include the dilemma of what business data should be stored and what should be discarded, the cost of acquiring needed technologies and a lack of IT professionals with the necessary skills.

The last problem will likely become more severe in the near future, according to Versace. "That gene pool is growing pretty shallow right now."

Some misperceptions also persist with regard to big data, Vesset said. For one thing, "it's not all about social media," he said. "That's one of the big fallacies out there, that big data is all about clickstream analysis."

Nor is Hadoop a sole solution, since it's oriented to large-scale batch processing and not real-time monitoring, such as a rail company tracking the performance of specific components in their train cars as they deliver shipments, he said.

The analyst firm has created a 'maturity model' for big data, which Vesset and Versace described in their session. It spans five subject areas: Data, people, process, technology and intent; and five stages of deployment: ad hoc, opportunistic, repeatable, managed and optimised.

A first step for companies just starting out with big data is to identify opportunities to use their existing technology and data in new ways, evaluate public cloud and open-source options, and start experimenting with proof-of-concept exercises and prototypes, according to the analysts.

Over the following one to two years, those companies should look to use early successes with big data projects to justify funding for larger efforts. In the same period, it would be wise to also look for sponsors within business departments who will champion big data projects, they said.

Some 80 per cent of what can be considered big data is unstructured or semi-structured information, said IDC analyst, David Schubmehl, during another presentation at the event. These sources could include anything from clickstream data to patent records, research archives and even video, he said.

This diversity will give rise to what IDC is calling unified information access technology, evidenced by products such as Oracle's Endeca and IBM's Vivisimo, as well as specialised vendors like Attivio.

Big data's challenges will also continue influencing the database industry, with growing prominence for technologies like graph and in-memory database platforms, said IDC analyst Carl Olofson during the presentation.

Traditional relational databases will also change, with capabilities expanding "to the point Ted Codd wouldn't recognise them," Olofson said, referring to the 'father' of the relational model.

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Show Comments

Blog Posts

3 skills you need to drive better collaboration

A study published in The Harvard Business Review found the time spent in collaborative activities at work has increased by over 50 per cent in the past two decades. Larger projects; complicated problems; tighter timeframes: These require bigger teams with specialised skillsets and diverse backgrounds, often dispersed globally.

Jen Jackson

CEO, Everyday Massive

Better the bank you know?

In 2018, only 21 per cent of customers believed that banks in general had their customers best interests at heart and behave ethically. Only 26 per cent believed that banks will keep their promises; views cemented further following the Hayne Financial Services Royal Commission.

Carolyn Pitt

Head of account management, Hulsbosch

What 15 years of emotional intelligence told us about youth media audiences

Taking people on an emotional journey through content is the most critical part of being a publisher. Which is why emotion lies at the heart of VICE Media.

Stephanie Winkler

Head of insights, VICE Asia-Pacific

This journey would identify all your future life aspects!

Maryann Humphrey

Open Colleges: one-to-one journeys is the goal

Read more

It's a pretty good idea. I think this integration is useful. Don't you agree?

Misty Stoll

Officeworks hops on voice interface bandwagon with Google Assistant integration

Read more

ok. so no RCS support? by the way, RCS is a lot bigger than 5G in terms of marketing and monetisation so y'all should be covering it.

DragoCubed

Optus goes for education with 5G network campaign

Read more

Many companies and individual merchants have shifted their major part of marketing to web marketing services Portland as it weighs fewer ...

Radiata Solutions

6 Ways to ramp up Social Media to Your Web Design

Read more

This is a unique experience! Will be interesting to talk to their managers.

Joyce Harris

​How Krispy Kreme revitalised its brand in a saturated market

Read more

Latest Podcast

More podcasts

Sign in