Episode II – Performance Specs and Benchmarks

Not so long ago, in an SSD company not very far away (just down the road, in fact), an astounding realization was made that NAND-based SSDs could achieve random access IOPS hundreds of times higher than Enterprise HDDs. Even more remarkable was the realization that Enterprise storage companies would pay high prices for those IOPS. Thus, began the "SSD IOPS Wars" that now consume the storage galaxy, as high performance Enterprise SSDs battle for supremacy in the data center.

This focus on high IOPS has taken on epic proportions. Not a week goes by without a new Enterprise solid state storage system claiming victory in the IOPS wars. Even with consumer grade SSDs, random read/write IOPS are often the primary performance specifications that users consider, that reviewers feature, and that suppliers tout. However, with this emphasis on IOPS performance, come several pitfalls. Here are some suggestions to avoid falling into a trap when comparing and benchmarking SSDs:

1. Make sure you're comparing apples to apples

Although the IOPS specifications of a Client grade SSD (cSSD) may appear to be similar to those of an Enterprise grade SSD (eSSD), they are architected for different types of data patterns and workloads, and their behavior can be drastically different under the same workload. cSSDs are typically optimized for PC applications where the drive is the primary storage and workloads are mainly large block reads. eSSDs are typically designed for enterprise applications such as a database server, serving multiple users with random access patterns. Therefore, it stands to reason that users should take care to compare SSDs of the same class when evaluating. Moreover, appropriate benchmarks should be used to evaluate different classes of drives.

2. Always read the fine print

On virtually all SSD product datasheets, you will find specifications for Sequential Read and Write Throughput (MB/s), and Random Read and Write IOPS. However, some vendors spec sustained performance, while others list maximum performance. The test conditions and parameters (usually in footnotes) will also vary from vendor to vendor, e.g.:

Transfer sizes – 4KB to 128KB

Queue depths (QD) – 1 to 64

Drive test area – 8GB LBA range to entire capacity of drive

Smaller transfer sizes (<16KB), a high QD, and a small drive test area serve to increase IOPS performance results. The same is true for sequential throughput tests, but with larger transfer sizes rather than smaller. The bottom line is that accurate comparisons cannot be made just by looking at datasheet specs.

3. Do your own benchmarking, but know what your benchmark is actually testing

IOMeter is a highly configurable I/O subsystem measurement tool, and IOMeter benchmarks have become a de facto standard by which all SSDs are compared. However, did you know that different versions of IOMeter give different results? In random write tests, IOMeter 2006 generates random data with higher entropy (or "randomness") than IOMeter 2008. This makes a huge difference when benchmarking SSDs that utilize data compression technology, which does not perform as well with high entropy data workloads. Understanding how different benchmarks function can help SSD users better assess whether they are relevant to their application.

Other Common Benchmarks:

• Atto® – Sequential read and write test with multiple transfer sizes
• PCMark Vantage – Approximates typical PC user workloads, e.g.
• AS-SSD – Sequential and Random R/W tests, with high entropy and multi-thread tests
• CrystalDiskMark - Sequential and Random R/W tests, with high entropy and multiple QD settings

4. Focus on relevant benchmark tests

Do 100% random 4KB IOPS benchmarks really matter? 100% random write tests only tell you how an SSD will perform in the worst case. But if one drive performs better than another under the worst case, it should also be better in lighter workloads, right? Not necessarily. Drive architectures are often optimized to do well in synthetic benchmark testing. An SSD that has 10% higher IOPS than another in a 100% random write IOMeter workload may turn out to have worse performance when a real world mixed workload is applied. High percentage read tests may be relevant for "cold storage" applications and streaming applications, but very rarely do tests set for 100% random/sequential or 100% read/write apply to real world workloads. Typical real world workloads will have a mix of reads and writes, so it is useful to benchmark R/W mixed workloads from 90/10, 80/20, 70/30, 60/40, to 50/50.

5. Always precondition before testing

It is well documented that some SSDs can suffer from performance degradation when the drive becomes full. Thus, “Fresh-Out-of-Box” (FOB) performance is not a good indicator of real world performance, because the SSD has not reached "steady-state", where it would perform at a sustained level after being used for some time in a real application. Preconditioning fills the drive with data so that it will trigger the background data management algorithms and garbage collection during write operations. Preconditioning is a necessary step to obtain more meaningful benchmark measurements.

6. Pay more attention to latencies than IOPS

If high IOPS are a priority for your application, chances are latencies will be too. It is in latency performance where one will find the most variation in SSD products. Low read/write latencies are important, but perhaps more important are consistently low latencies. Even the most advanced eSSDs with average latencies < 100us can encounter latency spikes into the hundreds of milliseconds. The key is to understand what percentage of I/O latencies will stay under a certain metric. In many enterprise applications, keeping a high percentage (~99%) of I/Os under 1ms latency would be considered excellent latency performance.

7. Application specific testing

Whenever possible, test in the actual system environment. Running a battery of benchmark tests can give you a picture of SSD performance in synthetic conditions relative to other drives, but there is no substitute for the real thing. However, when it is not practical to run in actual enterprise environments, the next best things are application specific simulator benchmarks such as the ones listed below, rather than the standard drive benchmark tools mentioned previously.

Jetstress 2013 – Simulates Microsoft Exchange Server disk I/O

Sysbench OLTP – System benchmark with true database-backed transactional test profile to MySQL

sql-bench – Single threaded benchmark suite provided with MySQL source distributions

Conclusion

What must we unlearn about SSDs? IOPS specifications and benchmark scores can be misleading and should not be the primary criteria when selecting an SSD. Look beyond the IOPS, do your own benchmarking, and strive to understand actual system performance under real workloads. Only then, will you continue your path to true mastery of the force known as SSD technology.

ATTO is a registered trademark of ATTO Technology, Inc.