Powering Facebook Open Graph, Apache Giraph was built from Yahoo and Google technologies
Move over Hadoop, there is another highly scalable data processing powerhouse in town: Apache Giraph. Facebook is using the technology to bring a new style of search to its billion users.
When Facebook built its Graph Search service, the social networking company picked Giraph over other social graphing technologies -- such as the Hadoop-based Apache Hive and GraphLab -- because of Giraph's speed and immense scalability.
"Analyzing these real-world graphs at our scale ... with available software was impossible last year. We needed a programming framework to express a wide range of graph algorithms in a simple way and scale them to massive datasets," wrote Facebook software engineer Avery Ching, in a blog post that discussed Facebook's use of the technology.
With a little modification, Facebook has used Giraph to analyze a trillion edges, or connections between different entities, in under four minutes.
In addition to using Giraph for its Graph Search, Facebook also plans to use the software for other duties such as targeting ads and ranking data.
Launched in January, Facebook's Graph Search service provides a way for users to query Facebook's massive collection of user-generated data and get back personalized results.
"Open Graph allows application developers to connect objects in their applications with real-world actions (such as user X is listening to song Y)," Ching explained.
Facebook's Graph Search, while still not as mature as regular search services such as Google's, may be the first widespread public introduction to the benefits of using social graphs.
A social graph maps the complex relationships between many different entities (called nodes). A node can be anything: a person, a restaurant, a city. They are connected by edges. An edge, for instance, asserts that a particular person may live in a certain city.
Yahoo first developed Giraph using the principles set forth in a 2010 paper published by Google engineers, "Pregel: a system for large-scale graph processing."
Using the Bulk Synchronous Parallel model of computing, Google designed Pregel to generate graphs from very large data sets, using lots of commodity servers.
Like it did with Hadoop, Yahoo bequeathed Giraph to the Apache Software Foundation, where it is now a fully open-source project worked on by developers from Facebook, LinkedIn, Twitter and Hortonworks.
Because Giraph is written in Java, Ching explained, it can connect very easily with the various parts of Facebook's Hadoop deployment, which it relies upon for data storage management and resource scheduling.
Facebook stores its user-generated data in a data warehouse running on Apache Hive, a component of Hadoop. Giraph, however, can generate graphs four times faster than Hive itself. Because it runs on Hadoop's MapReduce, a Giraph job can be split across multiple servers so it can be executed in parallel.
Facebook modified Giraph in a number of ways to make it run more efficiently, according to Ching.
Company engineers devised a number of tweaks to trim Giraph's memory usage on servers. "Giraph was a memory behemoth due to all data types being stored as separate Java objects," Ching wrote.
To improve Giraph's scalability, Facebook linked it with the Netty event-driven framework.
In one test using user interaction data, Facebook was able to use Giraph to create a 1 trillion-edge social graph in under four minutes, using 200 commodity servers.
Facebook's benchmark dwarfed previously published Giraph tests by other companies by at least two orders of magnitude. Heretofore, researchers have been able to create a 6.6 billion-edge graph using Yahoo Altavista data and a graph of Twitter data with 1.5 billion edges.
Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com
The current global COVID-19 pandemic is resulting in unprecedented disruption to every aspect of our lives as marketing and brand professionals, from the ways we work, to how our organisations operate, and the way in which we acquire and engage customers. So we’ve drawn on our wonderful CMO50 alumni community to explore different aspects of the crisis facing all of us right now.
The current world pandemic, COVID-19, and its tragic effects has created different and challenging situations for nearly every business. Every business sector is affected differently, depending on the nature of what your place in the world, creating the most unique situation most of us have ever and will ever experience during our professional lives.
People whose personal details and experiences signal they come from racially diverse backgrounds are less likely than anglo or Caucasian candidates to make it through the first cut in recruitment processes. Even if the organisation says it values diversity.
The impact of coronavirus is far reaching with the true impact on the economy and businesses is unknown. While there are a few categories and brands experiencing growth, for the most part the crisis is wreaking havoc for large and small operators across many sectors including entertainment, tourism, retail, fitness, services and the list goes on.
Why these voice assistants are so popular nowadays? Maybe I should get one too? I am really curious.
Jill Kim
Aussie brands jump on voice-interaction bandwagon following Amazon Alexa's local launch
Your page is very helpful. Thank you for sharing with us
Eriona Ajvazi
10 brands making a positive difference to a world in crisis
Extremely insightful and well written. Thanks for the great article!
Nicole Brodie Nahum
Why COVID-19 makes it more important than ever to move at the speed of the consumer
Blockchain is one of the fastest growing technology in today's digital era. Industries like banking and finance are already using blockch...
Aniket Singh
Can blockchain deliver on its big advertising promises?
Great article Emma. So many gems in there. Awesome to have you in the team!
One Small Step Collective
Why COVID-19 makes it more important than ever to move at the speed of the consumer