Hadoop is not enough for Big Data, says Facebook analytics chief

Don't discount the value of relational database technology, Ken Rudin told a Big Data conference

The Hadoop programming framework may be synonymous with the Big Data movement but it's not the only tool companies need to derive insights from massive stores of unstructured information, according to Facebook analytics chief, Ken Rudin.

"There are a lot of commonly held beliefs about big data that need to be challenged," with the first being that you simply adopt Hadoop and are good to go, Rudin said during a keynote at the Strata + Hadoop World conference in New York. "The problem is that Hadoop is a technology, and big data isn't about technology. Big Data is about business needs."

"In reality, Big Data should include Hadoop and relational [databases] and any other technology that is suitable for the task at hand," he added.

Facebook's business model depends on the way it crunches the streams of profile and activity data generated by the social media site's more than 1 billion users in order to deliver targeted advertisements. But "Hadoop isn't always the best tool for what we need to do," Rudin said.

For example, it makes sense to do broad exploratory analysis of a data set in Hadoop, but a relational store is better for conducting an operational analysis of what was uncovered, he said.

Hadoop is also good for looking at the lowest level of detail in a data set, but relational databases make more sense for storing transformed and aggregated data, Rudin added.

"The bottom line is, use the right technology for whatever it is you need," he said.

There's also the presumption that the mere act of analyzing big data provides valuable insights, Rudin said. "The problem is coming up with more brilliant answers to questions nobody cares about," he said. "It is still an art to figure out what the right questions are."

Facebook has focused on hiring the right staffers to run its analytics operations, people who not only have doctorates in statistics but are also business-savvy, he said.

"When you interview [candidates], don't focus just on, 'how do we calculate this metric,'" Rudin said. Instead, give them a business case study and ask them what would be the most important metrics to look at, he added.

Companies should also attempt to train "everyone on analytics", according to Rudin.

Facebook runs an internal "data camp," a two-week program that teaches employees about analytics. Product managers, designers, engineers and even finance department workers attend, Rudin said. "The value of having everybody go through it is, you give everybody a common language of data they can discuss problems and issues with," he said.

Facebook has also shaken up the way it organizes statisticians and business teams. If statisticians are kept separate, they tend to "sit there and wait for requests to come in from areas of the business and respond to them," instead of being proactive, he said.

But if statisticians are placed into business units, ""you'll find multiple groups trying to solve problems redundantly," he said.

Facebook has settled on an "embedded" model, wherein analysts are placed with business teams but report up to a higher-level group of analysts, which helps avoid duplicate efforts.

Chris Kanaracus covers enterprise software and general technology breaking news for The IDG News Service. Chris' email address is Chris_Kanaracus@idg.com

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Show Comments

Blog Posts

5 cornerstones of a strong digital culture

Creating a strong company culture may sound like a daunting task, but it’s actually pretty straightforward. In fact, company culture is created in exactly the same fashion as a religion or democracy. Behaviours created from the organisation’s inception are reinforced over time by leadership, attracting like-minded people and eventually reaching critical mass to become an accepted ‘truth’.

Anthony Stevens

Founder and CEO, Digital Asset Ventures

Should you rebuild your company’s tech stack in blockchain?

The question I get asked most regularly these days is: ‘Do I need to rebuild my company’s systems on Blockchain?’ And the answer, every time, is ‘No, you’re asking the wrong question’.

Michelle O'Keeffe

CEO, Engaging.io

Customer value proposition: Getting the brand promise to your customers right

Throughout my career, I have witnessed a litany of brand names that profess to have a unique customer value proposition (CVP). In reality, however, they’ve had little more than a ‘value proposition’: A simple list of benefits applied to a general audience.

Ric Navarro

Global director of marketing and communications, Norman, Disney & Young

Thanks for writing about chatbots. Definitely bots have the exciting future when it comes to customer engagement, transactional and conve...

Giridhar Prathap Reddy

Deloitte TMT Predictions: AR and mobile crucial for marketing success in 2018

Read more

Hi there! Thanks for this interesting article! I love to read about new technology and software that makes our lives easier. I'm looking ...

Julia Summer

Wartsila overhauls Web platforms to create ‘seamless’ brand experience across all devices

Read more

RE: Sales and marketing SLAs, often the choke point isn't the teams but them getting the data into the tools they want to use with the da...

Ed Fry

Why sales and marketing alignment is more important than ever

Read more

Thank you for the good and very helpful information. It is very interesting. I love all the things you share and see your beautiful creat...

รัตนาวดี ภูมิวรรณ

Former eHarmony marketing chief joins telco startup

Read more

Colin Kaepernick, not Mike Kapernick.

thisisw

Zenith's innovation leader: Mid-digital age not benefitting media, brands or consumers

Read more

Latest Podcast

More podcasts

Sign in