Yahoo has released a ton of anonymized user data to help machine learning scientists

The data includes about 110 billion interactions with Yahoo websites

Machine learning is taking hold in all kinds of applications, from self-driving cars to image recognition to online recommendation engines. But unless you're a Google or a Facebook, it's hard to get your hands on the kind of massive, real world data sets required to test and validate machine learning programs.

Yahoo has helped to rectifying that with the release Thursday of what it called the "largest ever" data set made available to machine learning scientists. It's a collection of anonymized user interactions with the news steams on sites like Yahoo News and Yahoo Sports.

Yahoo says there are 110 billion events in the file -- or 110 billion records of when a user clicked on a news story or took some other action in the feed -- and it comprises 13.5TB of data, or 1.5TB compressed. That's more than ten times the size of the previous largest dataset released, Yahoo says.

Yahoo News Feed Yahoo

The data comes from interactions with its news feed, the area in red above

"Data is the life-blood of research in machine learning," the company said. "However, access to truly large-scale datasets is a privilege that has been traditionally reserved for machine learning researchers and data scientists working at large companies -- and out of reach for most academic researchers."

Machine learning refers to a class of programs that "learn" and improve their ability to solve problems over time. An early example was spam detection, but machine learning is used for image recognition, language translation and a myriad of other tasks, including some for business. Google recently said it was "rethinking everything we're doing" around machine learning.

Computer scientists create models and write algorithms to guide machine learning systems, but they need large data sets on which to test those models and improve them.

They can use synthetic, artificially created data sets, but those don't reflect the messiness and unpredictable behavior that humans exhibit online, said Suju Rajan, Yahoo's director of research for personalization science.

"Real-world data is messy, it presents a lot of challenges, and those challenges aren't necessarily thought of when someone creates an artificial data set," she said. "If you don't take my behavior into account, the algorithm you create might not work so well."

She expects scientists to use the data to help build better recommendation engines, like those those on Netflix and Amazon. But she says it could also drive other research areas, like information retrieval, social feed ranking, and even systems engineering, by helping cloud providers decide how to process data as users interact with it.

The user data was due to be available for download Thursday through Yahoo Labs' Webscope data sharing program, a library of anonymized data sets for non-commercial use.

It's based on user interactions with Yahoo News, Sports, Finance, Movies, and Real Estate. The data was gathered over four months early last year from 20 million Yahoo users. In addition to the interaction data, it includes categorized demographic information, like age range and gender, for a subset of the users. It's also releasing the title, summary, and key-phrases of the related news articles.

Yahoo says the previous largest data set, released last year by the online marketing firm Criteo, was 1TB in size and included some 4 billion events.

It says its goal is to level the playing field a bit for academic researchers, who often have more freedom to pursue long range projects than their peers at corporations, but who lack the real world data to do it with.

"They might be able to solve problems in a way that we can make use of at Yahoo, or come up with new research problems that we haven't even thought of yet," Rajan said.

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Show Comments

Latest Videos

Conversations over a cuppa with CMO: Microsoft's Pip Arthur

​In this latest episode of our conversations over a cuppa with CMO, we catch up with the delightful Pip Arthur, Microsoft Australia's chief marketing officer and communications director, to talk about thinking differently, delivering on B2B connection in the crisis, brand purpose and marketing transformation.

More Videos

Great content and well explained. Everything you need to know about Digital Design, this article has got you covered. You may also check ...

Ryota Miyagi

Why the art of human-centred design has become a vital CX tool

Read more

Interested in virtual events? If you are looking for an amazing virtual booth, this is definitely worth checking https://virtualbooth.ad...

Cecille Pabon

Report: Covid effect sees digital events on the rise long-term

Read more

Thank you so much for sharing such an informative article. It’s really impressive.Click Here & Create Status and share with family

Sanwataram

Predictions: 14 digital marketing predictions for 2021

Read more

Nice!https://www.live-radio-onli...

OmiljeniRadio RadioStanice Uzi

Google+ and Blogger cozy up with new comment system

Read more

Awesome and well written article. The examples and elements are good and valuable for all brand identity designs. Speaking of awesome, ch...

Ryota Miyagi

Why customer trust is more vital to brand survival than it's ever been

Read more

Blog Posts

A Brand for social justice

In 2020, brands did something they’d never done before: They spoke up about race.

Dipanjan Chatterjee and Xiaofeng Wang

VP and principal analyst and senior analyst, Forrester

Determining our Humanity

‘Business as unusual’ is a term my organisation has adopted to describe the professional aftermath of COVID-19 and the rest of the tragic events this year. Social distancing, perspex screens at counters and masks in all manner of situations have introduced us to a world we were never familiar with. But, as we keep being reminded, this is the new normal. This is the world we created. Yet we also have the opportunity to create something else.

Katja Forbes

Managing director of Designit, Australia and New Zealand

Should your business go back to the future?

In times of uncertainty, people gravitate towards the familiar. How can businesses capitalise on this to overcome the recessionary conditions brought on by COVID? Craig Flanders explains.

Craig Flanders

CEO, Spinach

Sign in