Table of Contents
- Executive Summary
- Cloud IaaS SQL Server Offerings
- Field Test Setup
- Field Test Results
- Price Per Performance
- About Microsoft
- About William McKnight
- About Jake Dolezal
The fundamental underpinning of an organization is its transactions. They must be done well, with integrity and performance. Not only has transaction volume soared recently, but the level of granularity in the details has reached new heights. Fast transactions greatly improve the efficiency of a high-volume business. Performance is incredibly important.
There are a variety of databases available for transactional applications. Ideally, any database would have the required capabilities; however, depending on the application’s scale and the chosen cloud, some database solutions can be prone to delays. Recent information management trends see organizations shifting their focus to cloud-based solutions. In the past, the only clear choice for most organizations was on-premises data using on-premises hardware. However, the costs of scale have chipped away at the notion it is the best approach for some, if not all, a company’s transactional needs. The factors driving operational and analytical data projects to the cloud are many. Still, advantages like data protection, high availability, and scale are realized with infrastructure as a service (IaaS) deployment. In many cases, a hybrid approach serves as an interim step for organizations migrating to a modern, capable cloud architecture.
This report outlines the results from two GigaOm Field Tests (one transactional and the other analytic) derived from the industry-standard TPC Benchmark™ E (TPC-E) and TPC Benchmark™ H (TPC-H) to compare two IaaS cloud database offerings:
- Microsoft SQL Server on Amazon Web Services (AWS) Elastic Cloud Compute (EC2) instances.
- Microsoft SQL Server Microsoft on Azure Virtual Machines (VM).
Both are installations of Microsoft SQL Server 2019, and we tested Windows Server OS using the most recent versions available as a pre-configured machine image.
Data-driven organizations also rely on analytic databases to load, store, and analyze volumes of data at high speed to derive timely insights. Data volumes within modern organizations’ information ecosystems are rapidly expanding, placing significant performance demands on legacy architectures. Today, to fully harness their data to gain a competitive advantage, businesses need modern, scalable architectures and high levels of performance and reliability to provide timely analytical insights. In addition, many companies like fully managed cloud services. With fully managed as-a-service deployment models, companies can leverage powerful data platforms without the technical debt and burden of finding talent to manage the resources and architecture in-house. With these models, users only pay as they play and can stand up a fully functional analytical platform in the cloud with just a few clicks.
The results of the GigaOm Transactional Field Test are valuable to all operational functions of an organization, such as human resource management, production planning, material management, financial supply chain management, sales and distribution, financial accounting and controlling, plant maintenance, and quality management. The Analytic Field Test results are insightful for many of these same departments today using SQL Server, which is frequently the source for interactive business intelligence (BI) and data analysis.
Testing hardware and software across cloud vendors is challenging. Configurations favor one cloud vendor over another in feature availability, virtual machine processor generations, memory amounts, storage configurations for optimal input/output, network latencies, software and operating system versions, and the benchmarking workload. Our testing demonstrates a narrow slice of potential configurations and workloads.
Azure SQL Server 2019 Enterprise on Windows’ best transactions per second (tps) was 42% higher than AWS SQL Server 2019 Enterprise on Windows. Azure SQL Server 2019 Enterprise on Windows’ best queries per hour (QPH) had 41% higher QPH than AWS SQL Server 2019 Enterprise on Windows Server.
When it comes to transaction processing, Azure’s price-performance is 23% less expensive than the price-performance of AWS SQL Server on Windows without license mobility. With license mobility, Azure SQL Server on Windows provided price-performance that was 27% less expensive than AWS. Azure price-performance is almost 31% less expensive than the price-performance of AWS SQL Server on Windows with license mobility and a three-year commitment.
For analytic processing, the price-performance of Azure SQL Server on Windows without license mobility proved to be 21% less expensive than AWS. With license mobility in place, the price-performance advantage for Azure widened to 23%. And for Azure SQL Server on Windows with license mobility and a three-year commitment, price-performance was 25% less expensive than AWS.
As the report sponsor, Microsoft selected the particular Azure configuration it wanted to test. GigaOm selected the closest AWS instance configuration for CPU, memory, and disk configuration.
We leave the issue of fairness for the reader to determine. We strongly encourage you to look past marketing messages and discern what is of value. We hope this report is informative and helpful in uncovering some of the challenges and nuances of platform selection.
In the same spirit of the TPC, price-performance intends to be a normalizer of performance results across different configurations. Of course, this has its shortcomings, but at least one can determine “what you pay for and configure is what you get.”
The parameters to replicate this test are provided. We used the BenchCraft tool, audited by a TPC-approved auditor who reviewed all updates to BenchCraft. All the information required to reproduce the results are documented in the TPC-E specification. BenchCraft implements the requirements documented in Clauses 3, 4, 5, and 6 of the benchmark specification. There is nothing in BenchCraft that alters the performance of TPC-E or this TPC-E-derived workload.
The scale factor in TPC-E is defined as the number of required customer rows per single tpsE. We changed the number of initial trading days (ITD). The default value is 300, which is the number of eight-hour business days to populate the initial database. For these tests, we used an ITD of 30 days rather than 300. This reduces the size of the initial database population in the larger tables. The overall workload behaves identically with ITD of 300 or 30 as far as the transaction profiles are concerned. Since the ITD was reduced to 30, any results would not be compliant with the TPC-E specification and, therefore, not comparable to published results. This is the basis for the standard disclaimer that this is a workload derived from TPC-E.
However, BenchCraft is just one way to run TPC-E. All the information necessary to recreate the benchmark is available at TPC.org (this test used the latest version, 1.14.0). Just change the ITD, as mentioned above.
We have provided enough information in the report for anyone to reproduce these tests. You are encouraged to compile your own representative queries, data sets, data sizes, and test compatible configurations applicable to your requirements.