Databricks takes on Google data streaming analysis with Spark

Databricks Cloud will provide Spark-based streaming analysis as a service

Taking on Google, Databricks plans to offer its own cloud service for analyzing live data streams, one based on the Apache Spark software.

Databricks Cloud is designed to provide a platform for analyzing streaming data, much like the Google DataFlow service announced last week.

Like Google DataFlow, Databricks Cloud promises to offer a single programming model that cuts across different approaches to data analysis, including support for batch programming and live data streaming. And like Google DataFlow, Databricks Cloud will first be offered in preview mode, with full commercial support due by the end of the year.

The two services are aimed to different markets, according to Ion Stoica, CEO of Databricks.

"Google DataFlow is really targeted to developers. We also have higher-level interfaces for data scientists and data engineers," Stoica said.

Databricks also guarantees application portability. Because the entire stack is based on open source software, users can move their workloads to other Apache Spark installations should they need to, Stoica said. "You can take your application and run it in another cloud," Stoica said.

Such a service could be used by enterprises for tasks such as churn analysis, which can determine why a customer stops using a product, or for fraud detection, where a malicious activity can be spotted while it is still taking place.

The University of California, Berkeley's AMP (Algorithms, Machines and People) Lab originally developed Spark as a unified processing engine, one able to provide a platform for a variety of data analysis tasks, including interactive queries, steaming data analysis, machine learning and graph computation.

A number of developers behind Spark went on to form Databricks. The software itself, designed to run on a cluster of servers, is now managed as an open source project under the guidance of the Apache Software Foundation.

Offering Spark as a service eliminates the arduous task for setting up and maintaining an in-house implementation of Spark, Stoica noted.

"Clusters are hard to set up and maintain. To build a data pipeline, you need to stitch together multiple tools, and the tools are still hard to use. So extracting value out of the data is still a struggle," Stoica said.

Initially, Databricks Cloud will be run on Amazon Web Services, though eventually it will also run on other cloud providers such as Google.

In addition to the Spark platform itself, Databricks will provide a set of built-in applications that can do common data analysis tasks. Users can build their own workflows, or issue queries and interact with the data directly. Output can be piped to a dashboard or a report.

Databricks is not the only company making use of Spark's capabilities. ClearStory offers an analytics software package based on Spark that allows organizations to aggregate dozens of unstructured data sources for analysis, far more than can be easily done through traditional business intelligence tools, said ClearStory CEO Sharmila Mulligan.

Databricks also announced Monday that it has received US$33 million in series B funding led by venture capital firm, New Enterprise Associates, with follow-on investment from Andreessen Horowitz.

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Show Comments

Latest Videos

More Videos

Great piece Katja. It will be fascinating to see how the shift in people's perception of value will affect design, products and services ...

Paul Scott

How to design for a speculative future - Customer Design - CMO Australia

Read more

Google collects as much data as it can about you. It would be foolish to believe Google cares about your privacy. I did cut off Google fr...

Phil Davis

ACCC launches fresh legal challenge against Google's consumer data practices for advertising

Read more

“This new logo has been noticed and it replaces a logo no one really knew existed so I’d say it’s abided by the ‘rule’ of brand equity - ...

Lawrence

Brand Australia misses the mark

Read more

IMHO a logo that needs to be explained really doesn't achieve it's purpose.I admit coming to the debate a little late, but has anyone els...

JV_at_lAttitude_in_Cairns

Brand Australia misses the mark

Read more

Hi everyone! Hope you are doing well. I just came across your website and I have to say that your work is really appreciative. Your conte...

Rochie Grey

Will 3D printing be good for retail?

Read more

Blog Posts

How to design for a speculative future

For a while now, I have been following a fabulous design strategy and research colleague, Tatiana Toutikian, a speculative designer. This is someone specialising in calling out near future phenomena, what the various aspects of our future will be, and how the design we create will support it.

Katja Forbes

Managing director of Designit, Australia and New Zealand

The obvious reason Covidsafe failed to get majority takeup

Online identity is a hot topic as more consumers are waking up to how their data is being used. So what does the marketing industry need to do to avoid a complete loss of public trust, in instances such as the COVID-19 tracing app?

Dan Richardson

Head of data, Verizon Media

Brand or product placement?

CMOs are looking to ensure investment decisions in marketing initiatives are good value for money. Yet they are frustrated in understanding the value of product placements within this mix for a very simple reason: Product placements are broadly defined and as a result, mean very different things to different people.

Michael Neale and Dr David Corkindale

University of Adelaide Business School and University of South Australia

Sign in