How Storage Compute Separation is Changing the Way Enterprises Interact with Their Data
By Justin Borgman
I’m sure you know the difference between storage and compute, and why the separation of these two layers is such a critical piece of an enterprise’s move to the cloud. Everyone here at Starburst Data is very familiar with the subject, too. So they call me “Captain Obvious” because I still insist on explaining it to prospective customers, IT leaders, Business Intelligence analysts, and anyone else who will listen.
But the separation of storage and compute is a tremendously important development. It’s core to what we’re doing with Starburst Presto here at Starburst Data and it is truly changing the way businesses interact with their data.
So please excuse me while I explain it once more.
The Old Model of Maintaining Resources
Starburst Presto completely separates storage and compute. You can leave your data in the cheapest storage layer, then spin up compute resources when you need them, and only for as long as you need them. You only pay for compute when you’re actually running your analytics…
Before the cloud this was not possible. You’d buy all your hardware upfront, along with the associated licenses and service contracts, and stock your data center with all the resources you might need to store and analyze your data.
If your peak usage required 100 machines, then you’d buy 100 machines – even if you only needed all those resources for a few hours day. The rest of the time, your expensive hardware would be sitting dormant, rapidly depreciating.
Those costs were never recouped. They were sunk capital expenditures amounting to wasted money.
The Shift to Elastic Cloud Economics
The cloud allows enterprises to pay only for what they use. This applies across cloud technologies and Starburst Presto lets companies take advantage of this new arrangement for data processing and data analysis.
First, you store all your data in affordable cloud storage, such as Amazon S3, Microsoft’s Azure Data Lake, or Google Cloud Storage. Then, when you want to process this data, you spin up virtual machines to do the work, but only for as long as you need them.
In the past, people assumed that you needed to have your data in a traditional database to run high-performance queries. But the open-source formats common to the cloud, such as ORC and Parquet, mimic the performance of some of the fastest databases. Query engines like Presto are getting faster and faster, so you can benefit from the cost savings of inexpensive cloud storage without sacrificing performance or results.
How Major Retailers Benefit from the Shift
Let’s say you’re a large retailer. Every morning, the CEO or Executive Team wants a complete report on every product sold the day before, at every store across the country. This isn’t a quick job. It might take a few hours, and in the past you’d need to have the hardware resources capable of querying all your databases and churning through this data to generate those results in a few hours. The other 22 hours of the day? Those expensive resources would sit idle, depreciating.
Here’s where the separation of storage and compute becomes so important. Today, this data is stored in different databases or cloud storage, and you pay one price for that. A lower one if you pick the cloud.
Starburst Presto allows you to query all your data where it resides, no matter where it lives, and spin up compute resources as needed. Instead of stocking and maintaining a data center, you merely create a batch reporting cluster that runs every night on a schedule. You dial up the necessary resources for as long as it takes to complete the work, then dial them down. You can even have different clusters for different tasks (batch reporting vs interactive BI) or different departments (marketing vs R&D). This allows for clean resource isolation and can even help with chargebacks and budgeting.
The Executive Team still gets its daily report, but the company pays much less to generate that analysis.
Retail is just one example. We’re seeing these kinds of results across industries, from the financial services sector to healthcare. Reach out to us today to find out how we can help you get more out of your enterprise data.
Want to learn more?
For more information about the latest version of Starburst Presto and how it can help your organization gain more access and control of your data, please complete this form or click here to read more.