Databricks, the data and AI company, announced the launch of SQL Analytics, which for the first time enables data analysts to perform workloads previously meant only for a data warehouse on a data lake. This expands the traditional scope of the data lake from data science and machine learning to include all data workloads including Business Intelligence (BI) and SQL. Now, organizations can empower data teams across data engineering, data science, and data analytics to work on a single source of truth for data. SQL Analytics realizes Databricks’ vision for a lakehouse architecture that combines data warehousing performance with data lake economics, resulting in up to 9x better price/performance than traditional cloud data warehouses. SQL Analytics is available in public preview effective on November 18.
A lakehouse architecture simplifies data and AI for organizations. In the past, data teams had to maintain proprietary data warehouses for business intelligence workloads and data lakes for data science and machine learning workloads, because no single data platform could meet the performance needs of BI and the flexibility needs of data science. Expensive and complicated to maintain, this coexistence of legacy architectures has created data silos that slow innovation and stifle data team productivity. A lakehouse addresses this by running all workloads through a single architecture.
Shell chose Databricks to be one of the foundational components of its Shell.ai platform. “Shell has been undergoing a digital transformation as part of our ambition to deliver more and cleaner energy solutions. As part of this, we have been investing heavily in our data lake architecture. Our ambition has been to enable our data teams to rapidly query our massive datasets in the simplest possible way. The ability to execute rapid queries on petabyte scale datasets using standard BI tools is a game changer for us. Our co-innovation approach with Databricks has allowed us to influence the product roadmap and we are excited to see this come to market.” Dan Jeavons, GM Data Science
“It is no longer a matter of if organizations will move their data to the cloud, but when. A lakehouse architecture built on a data lake is the ideal data architecture for data-driven organizations and this launch gives our customers a far superior option when it comes to their data strategy,” said Ali Ghodsi, CEO and co-founder of Databricks. “We’ve worked with thousands of customers to understand where they want to take their data strategy, and the answer is overwhelmingly in favor of data lakes. The fact is that they have massive amounts of data in their data lakes and with SQL Analytics, they now can actually query that data by connecting directly to their BI tools like Tableau.”
SQL Analytics is built on Delta Lake, an open format data engine that adds reliability, quality, and security, to a customer’s existing data lake. Customers are able to avoid storing multiple copies of data, as well as locking data up in proprietary formats. To deliver BI-performance on a data lake, SQL Analytics makes use of two unique innovations. First, it provides easy-to-use auto-scaling endpoints that keep query latency consistently low under high user load. Second, it uses Delta Engine, Databricks’ unique query execution engine, to complete queries extremely quickly against both large and small data sets. With native connectors for all major BI tools, including Tableau and Microsoft Power BI, customers can easily integrate SQL Analytics into their existing BI workflows to conduct analytics on much fresher, more complete data than ever before. SQL Analytics also provides a SQL-native query and visualization interface to allow analysts, data scientists, and developers without access to traditional BI tools to build dashboards and reports that can be easily shared within their organization.
“Now more than ever, organizations need a data strategy that enables speed and agility to be adaptable,” said Francois Ajenstat, Chief Product Officer at Tableau. “As organizations are rapidly moving their data to the cloud, we’re seeing growing interest in doing analytics on the data lake. The introduction of SQL Analytics delivers an entirely new experience for customers to tap into insights from massive volumes of data with the performance, reliability and scale they need. We’re proud to partner with Databricks to bring that opportunity to life.”
The lakehouse architecture is widely supported by Databricks partners including:
- BI Partners: Tableau, Power BI, Qlik, Looker, Thoughtspot
- Ingest Partners: Fivetran, Fishtown Analytics, Matillion, Talend
- Catalog Partners: Collibra, Alation
- Consulting Partners: Slalom, Thorogood, Advancing Analytics
“Databricks SQL Analytics is a critical step in the most important trend in the modern data stack: the unification of traditional SQL analytics with machine-learning and data science,” said George Fraser, CEO at Fivetran. “Companies make huge investments in centralizing and curating data, and they should be able to make those investments once and then implement multiple analytical paradigms in a unified environment. The Lakehouse architecture supports that.”
Sign up for the free insideAI News newsletter.
Speak Your Mind