Improving query performance and focusing on insights while reducing costs

About

This business communication platform is an essential component of daily business for many industries. The platform effectively replaces email within organizations, allowing employees to exchange messages, share files, and collaborate individually or as teams. The company manages a large amount of data, including 270 million messages, 215 billion logs, and 700 billion records every day.

The company stores its data entirely in the Amazon Simple Storage Service (Amazon S3) and uses Amazon EMR to query the data. As the company grew, it increased its usage of AWS and third-party services. This created scalability issues and made it necessary to upgrade its query environment.

Challenge

The company was spending a lot of money on compute but not getting the valuable insights it wanted. There were 20,000 queries per day, but the success rate was volatile, fluctuating between 50-90%. As a result, analysts often had to re-run queries over and over to get them to work. 

These failed queries during peak usage times drove up Amazon Elastic Compute Cloud (EC2) costs, while undermining confidence among various internal teams looking to glean insights from the data. 

The company had a single cluster and tried to run every query on it, which contributed to the high failure rate. As the company grew, it got to the point where it couldn’t scale the cluster anymore, and adding an extra node didn’t help.  

Solution

The company chose Starburst Enterprise to power fast, interactive queries with high concurrency. This allowed them to deliver actionable customer insights and improved business agility.

The company saw an increase in query success rate and cluster uptime, which led to a reduction in resource costs as well. Starburst’s cost-based optimizer automatically optimized queries, and the data engineering team was able to further tune its network. 

Because Starburst Enterprise gave the company the ability to separate storage from compute, the company was able to deploy multiple clusters. This solution allowed the data engineering team to optimize one cluster for fast, simple queries, while optimizing the other cluster for more complex queries and more simultaneous users.

Results

Deploying Starburst Enterprise resulted in immediate improvements for the company. 

• The business intelligence team now sees a query success rate above 99%.
• The team no longer has to run queries multiple times.
• Starburst’s cost-based optimizer ensures that queries are as fast and efficient as possible.
• Improved queries and efficiency led to fewer queries and a 20% reduction in EC2 costs.

As the company prepared for its Initial Public Offering, Starburst’s integration with Apache Ranger helped demonstrate the company’s commitment to strong security. This partnership gives the data engineering team fine-grained control, including column-level masking and row-level filtering that limits employee access to sensitive data.

The success of Starburst Enterprise has prompted the company to center its business intelligence operations around the platform. The company’s partnership with Starburst also gives the company input into future features and enhancements, along with enterprise-grade support from leading Trino experts.