Twitter Boosts Performance and Cost Efficiency

Twitter increases Hadoop performance and cost efficiency with caching, fast SSDs and more compute.

Executive Overview
Storage I/O can be a significant performance bottleneck for Hadoop* clusters, especially in hyperscale deployments like those at Twitter, where a single cluster can have up to 10,000 nodes and nearly 100 PB of logical storage. The typical Hadoop cluster at Twitter contains over 100,000 hard disk drives (HDDs)—but this configuration was reaching an I/O performance limit because while HDD capacity has increased over time, HDD performance has not significantly changed.2 Therefore, simply adding more, bigger HDDs wasn’t going to solve Twitter’s scaling challenges—in fact, it would make things worse as the I/O per GB decreases. Adding more spindles per node was not feasible due to space and power limitations.

Working in collaboration with an Intel engineering team, Twitter engineers conducted a series of experiments that revealed that storing temporary files managed by YARN* (Yet Another Resource Negotiator*) on a fast SSD enabled significant performance improvements on existing hardware (up to a 50 percent reduction in runtime).3 The team also discovered that removing a storage I/O bottleneck enabled them to use larger hard drives while simultaneously increasing processor utilization, which in turn resulted in the ability to use higher-core-count processors. This positively affected storage performance, and contributed to higher data center density by reducing the number of required HDDs.

Higher density leads to total cost of ownership (TCO) savings through energy efficiency, fewer racks, and a smaller data center footprint. Overall, Twitter expects that caching temporary data and increasing core counts will result in approximately 30 percent lower TCO and over 50 percent faster runtimes, compared to their legacy production cluster configuration.1

Read the white paper - Boosting Hadoop* Performance and Cost Efficiency with Caching, Fast SSDs, and More Compute

Explore Related Products and Solutions

Intel® Xeon® Scalable Processors

Drive actionable insight, count on hardware-based security, and deploy dynamic service delivery with Intel® Xeon® Scalable processors.

Learn more

Intel® SSD DC Series

Intel® SSDs for the data center are optimized for performance, reliability, and endurance.

Learn more

Notices and Disclaimers

Intel® technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at https://www.intel.co.uk. // Software and workloads used in performance tests may have been optimized for performance only on Intel® microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit https://www.intel.co.uk/benchmarks. // Performance results are based on testing as of the date set forth in the configurations and may not reflect all publicly available security updates. See configuration disclosure for details. No product or component can be absolutely secure. // Cost reduction scenarios described are intended as examples of how a given Intel®-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. // Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. // In some test cases, results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance.

Product and Performance Information

1

Baseline: Single-socket Intel® Xeon® E3-1230 processor v6 (4 cores); 32 to 64 GB RAM; 1x 1 TB or 2 TB HDDs; Intel S4500 240 GB boot disk; 1 GbE to 10 GbE Ethernet; no caching. Test: Single-socket Intel® Xeon® Gold 6262 processor (24 cores); 192 GB RAM; Intel S4500 240 GB boot disk; 8x 6 TB HDDs; 1x Intel® SSD DC P4610 6.4TB; 25 GbE Ethernet; caching using Intel® Cache Acceleration Software (Intel® CAS). OS: Twitter CentOS* 6 Derivative, Kernel Version 2.6.74-t1.el6.x86_64 (based on upstream 4.14.12 Kernel), BIOS Version: D3WWM11, Microcode Version: 0xb000021.

2

Backblaze, September 2018, “Hard Disk Drive (HDD) vs Solid State Drive (SSD): What’s the Diff?” https://www.backblaze.com/blog/hdd-versus-ssd-whats-the-diff/.

3

Baseline: Dual-socket Intel® Xeon® E5-2630 processor v4 @ 2.2 GHz (10 cores/20 threads per socket); 128 GB RAM; 12x 6 TB 7200 RPM SATA HDD; 1x SATA SSD boot disk; 25 GbE Ethernet; 102 nodes spread across 6 racks. Workload: Gridmix* and Terasort*. Gridmix Score: 3309 seconds; Terasort Score: 5504 seconds Test: Dual-socket Intel® Xeon® E5-2630 processor v4 @ 2.2 GHz (10 cores/20 threads per socket); 128 GB RAM; 12x 6 TB 7200 RPM SATA HDD; 1x SATA SSD boot disk; 1x 750 GB Intel® Optane™ DC P4800X NVMe*-based SSD; 25 GbE Ethernet; 102 nodes spread across 6 racks. Workload: Gridmix and Terasort. Gridmix Score: 2396 seconds; Terasort Score: 2640 seconds OS: Twitter CentOS* 6 Derivative, Kernel.