Comcast is one of the nation’s leading providers of communications, entertainment and cable products and services. Headquartered in Philadelphia, PA, they employ over 100,000 employees nationwide whose goal is to deliver the highest level of service and improve the customer experience. Comcast decided to run what they call their “ Hadoop Query Performance Smackdown ” for SQL engines.
The Comcast Big Data team uses an enterprise data like with over 1000+ daily active users running on 70 racks with petabytes of usable enterprise data available via Hive tables. Their uses cases range from ad hoc jobs, batch and streaming data, and reporting. They wanted to pick a SQL engine which would give them the best performance for the most practical use cases. They ran tests against MapReduce, Spark, LLAP, Tez and Presto. The end result was to pick a SQL engine to recommend to the CTO and the community.
They used a test methodology which utilized TPC-DS queries defined in the Hive benchmark for each of the SQL engines. Each query was run one at a time to utilize all the resources from the cluster. The team ensured that care was taken to tune and configure the engines. Furthermore, each query was run three times to make sure there were no anomalies. By doing this, the team calculated an average run time from the three rounds. Take notice of how the tests that were run against LLAP have much faster execution times that the other engines. LLAP had been explained as only been best optimized for ORC. The Comcast team found that they achieved much better performance across the board. LLAP was by far the fastest engine against Tez and Presto. SparkSQL did not manage to complete the benchmark successfully.
*SQL and Hadoop Query Performance Smackdown
To learn more, read the Datanami article.
In addition, watch the Comcast youtube video session from the Hortonworks Dataworks Summit on June 14, 2017 in San Jose, CA to learn about how you can use these results to help guide your company’s big data initiative on that journey of supporting interactive queries.
For more information on Apache Hive Performance Booster with LLAP, read our bloghere.