Facebook has found that Giraph scales near linearly with the number of workers or the problem size.
Move over Hadoop, there is another highly scalable data processing powerhouse in town: Apache Giraph. Facebook is using the technology to bring a new style of search to its billion users.
When Facebook built its Graph Search service, the social networking company picked Giraph over other social graphing technologies -- such as the Hadoop-based Apache Hive and GraphLab -- because of Giraph's speed and immense scalability.
"Analyzing these real-world graphs at our scale ... with available software was impossible last year. We needed a programming framework to express a wide range of graph algorithms in a simple way and scale them to massive datasets," wrote Facebook software engineer Avery Ching, in a blog post that discussed Facebook's use of the technology.
With a little modification, Facebook has used Giraph to analyze a trillion edges, or connections between different entities, in under four minutes.
In addition to using Giraph for its Graph Search, Facebook also plans to use the software for other duties such as targeting ads and ranking data.
Launched in January, Facebook's Graph Search service provides a way for users to query Facebook's massive collection of user-generated data and get back personalized results.
"Open Graph allows application developers to connect objects in their applications with real-world actions (such as user X is listening to song Y)," Ching explained.
A social graph maps the complex relationships between many different entities (called nodes). A node can be anything: a person, a restaurant, a city. They are connected by edges. An edge, for instance, asserts that a particular person may live in a certain city.
Using the Bulk Synchronous Parallel model of computing, Google designed Pregel to generate graphs from very large data sets, using lots of commodity servers.
Like it did with Hadoop, Yahoo bequeathed Giraph to the Apache Software Foundation, where it is now a fully open-source project worked on by developers from Facebook, LinkedIn, Twitter and Hortonworks.
Because Giraph is written in Java, Ching explained, it can connect very easily with the various parts of Facebook's Hadoop deployment, which it relies upon for data storage management and resource scheduling.
Facebook stores its user-generated data in a data warehouse running on Apache Hive, a component of Hadoop. Giraph, however, can generate graphs four times faster than Hive itself. Because it runs on Hadoop's MapReduce, a Giraph job can be split across multiple servers so it can be executed in parallel.
Facebook modified Giraph in a number of ways to make it run more efficiently, according to Ching.
Company engineers devised a number of tweaks to trim Giraph's memory usage on servers. "Giraph was a memory behemoth due to all data types being stored as separate Java objects," Ching wrote.
To improve Giraph's scalability, Facebook linked it with the Netty event-driven framework.
In one test using user interaction data, Facebook was able to use Giraph to create a 1 trillion-edge social graph in under four minutes, using 200 commodity servers.
Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com
Many marketing leaders are scrambling to give their people more flexible work. And they need to: PWC’s Future of Work report showed 90 per cent of staff want some form of hybrid or remote workplace.
Your success as a senior marketing professional has much in common with your success as a leader. Both marketing, and leadership activities, depend on building trust, encouraging action, and reliably fulfilling promises that have been made.
Between further interest rate rises, inflation, empty shelves, extortionate lettuce prices, supply chain issues and the barely believable events in Eastern Europe, the past six months there’s been a cacophony of environmental factors.
Invest and earn with Coinbloc .us. Guaranteed Weekly ROI, early signals, fast withdrawals among others. I recommend Coinbloc .us as on...
Hans Jensen
Explainer: What marketers need to know about cryptocurrency
Investment decisions are a big deal, so why not get some guidance? You can day-trade cryptos, BUY and HOLD and evaluate the assets with f...
Dave Sigurd
Gartner: Digital isn't enough of a superpower for CMOs anymore
I normally don’t feel comfortable investing online but because the company I worked for downsized due to the pandemic and I was one of th...
Dave Sigurd
CMO's top 8 martech stories for the week - 9 June 2022
Investment decisions are a big deal, so why not get some guidance? You can day-trade cryptos, BUY and HOLD and evaluate the assets with f...
Dave Sigurd
Creating a marketplace for wellness
A solution for an retail industry data extraction. https://e-scraper.com/usefu...
"e-Scraper" Data Extracting
Catchoftheday launches fee-based online shopping club