Menu

Rack Cooling, Workload Management Tools Fight Hotspots

July 2, 2014

Read this story at TechTarget.com
Hotspots are a pain in the CRAC, leading some data centers to implement source-of-heat cooling and software tools that redistribute workloads across servers. MGHPCC is in the vanguard.


 
Some IT teams push cabinet densities to reach peak efficiency per square foot and build cooling structures to support that. For others, even without hitting high kW densities per cabinet, IT departments run into hotspots.
Some like it hot
The research computing pods at the Massachusetts Green High Performance Computing Center (MGHPCC) in Holyoke contain petabytes of storage, high-speed central processing units (CPUs) and general processing units (GPUs) in blade servers, linked together via InfiniBand networks.
“We’re running systems hot, hot, hot all the time — the goal is 100% efficiency,” said James Cuff, researcher at Harvard University who spoke about the center during the AFCOM Symposium 2014 in Boston last month.
Cabinets at the MGHPCC are designed for a typical 14 kW per rack load, and average around 10 kW per rack. However, they’re capable of 20 or 25 kW per rack easily, and 100 kW density is theoretically possible.
We’re running systems hot, hot, hot all the time — the goal is 100% efficiency.” James Cuff, researcher, Harvard University
“As power grows, water — liquid cooling — is definitely coming back into the data center,” Cuff said. “But right now, the price/performance ratio favors air cooling.”
MGHPCC uses hot aisle containment and in-row cooling to maintain 81 degrees Fahrenheit inlet temperature with a temperature delta of 25 degrees to a 106 degree Fahrenheit hot aisle, according to James Culbert, IT technical lead at the center.
Dense cabinets create complex air flow patterns that demand attentive layout and rigorous cooling. A data center is like a game of Tetris, said Hassan Moezzi, CEO of Future Facilities Ltd., a London-based provider of computational fluid dynamics modeling software and services.
While the goal is 100% packed racks, “in reality, players create voids and gaps,” he said. “Physical fragmentation destroys energy efficiency.” One Future Facilities client ended up with overheating racks even at 45% capacity.
The MGHPCC arranges racks to keep the cabling out of the way of heat flow, but with high-density and mixed-use racks, they still end up with hotspots that are cooled by the in-row systems. Other data centers take cooling even closer to the source of heat.
“In-rack cooling could save us a lot of money, effectively turning off our ACs, but we haven’t pulled the trigger on it yet,” said Greg Tupper, IT services manager at MTS Systems Corp., a testing and sensing company headquartered in Minnesota.
MTS has pulled out a lot of old servers, so they’d need to re-rack to increase density. Tupper envisions each rack at least 70% filled — before putting in the coolers. Rack cooling vendors offer top-down or rear-door models, and various configuration options. Tupper advised investigating if your existing racks are compatible with a cooling system vendor’s products — not every rack is the same size or depth.
Rack cooling systems installation is a large one-time capital expense, with associated labor and design costs. Tupper recommended doing cost comparisons on four or five vendors, going in with solid knowledge about your density, existing problems and goals.
“For example, I like the modularity of the radiator units in a door that OptiCool [Technologies] offers, and the product is multipathing in case of a failure,” Tupper said. “The top-down design from Subzero [Engineering] wouldn’t work with the setup we currently have in the space.”
MGHPCC found that they overdid the initial in-row cooler specifications, and were able to take out several coolers from each installation without degrading cooling performance.
“That’s something we worry about too, just blanketing the rack and over-investing,” Tupper said.
However, the drive to higher-density cabinets hasn’t picked up speed as much as expected, according to William Dougherty, SVP and CTO of RagingWire Data Centers, a colocation provider in the U.S.
The majority of colocation customers and providers fill racks to the 4 to 6 kW per cabinet range, he said. “Very few customers are pushing over 10 kW per cabinet.”
Dougherty believes the increased energy efficiency of processors keeps most commercial servers and IT equipment from pulling as much power as previous generations.
“Users aren’t seeing the benefit of going more dense,” he said, so it’s pointless to increase density and deal with the specialized cooling requirements.
Workload management alleviates hotspots
Instead, data center IT staff can ameliorate hotspots by redistributing workloads, said Dave Wagner, director of market development at TeamQuest Corp., a systems management software provider based in Iowa. There are probably underutilized servers in another rack that could take on workloads from a hot cabinet, Wagner said. The trick is to know where you have room and where you’re overtaxing the silicon.
Workload management is “a whole lot cheaper than new CRACs,” Wagner said.
Your typical enterprise has many different form factors from 17 different vendors and several product generations in a typical data center, he added. You must build a map of the physical — heat and power draw — as well as the workload distribution — where applications are consuming resources. Data center infrastructure management tools are getting there, he said, and they require IT and facility teams to both look at this map, and look often.
After all, the data center is full of IT equipment that operates on one constant: change.
“You think you’ve designed in flexibility and can do whatever you want power-wise as long as it is below your max number of, for example, 2 megawatts,” Future Facilities’ Moezzi said. “But every change erodes capacity and flexibility.”
Anyone retrofitting or building a new data center should accurately gauge per-cabinet power draw to prevent over cooling or underutilizing cooling equipment and running it at low efficiency. As Wagner said, baseline everything you can. But under heavy load, cabinets might pull five times more power than under a typical load, so how do you design for both? Leave room to spread the load.
An automation and orchestration layer like PowerAssure’s Software Defined Power lets data centers move workloads based on operational requirements, RagingWire’s Dougherty said. When demand fluctuates, the important thing is to spin servers up here and down there reliably and consistently, which pays off in terms of power use, latency and management, he said.

Research projects

A Future of Unmanned Aerial Vehicles
Yale Budget Lab
Volcanic Eruptions Impact on Stratospheric Chemistry & Ozone
The Rhode Island Coastal Hazards Analysis, Modeling, and Prediction System
Towards a Whole Brain Cellular Atlas
Tornado Path Detection
The Kempner Institute – Unlocking Intelligence
The Institute for Experiential AI
Taming the Energy Appetite of AI Models
Surface Behavior
Studying Highly Efficient Biological Solar Energy Systems
Software for Unreliable Quantum Computers
Simulating Large Biomolecular Assemblies
SEQer – Sequence Evaluation in Realtime
Revolutionizing Materials Design with Computational Modeling
Remote Sensing of Earth Systems
QuEra at the MGHPCC
Quantum Computing in Renewable Energy Development
Pulling Back the Quantum Curtain on ‘Weyl Fermions’
New Insights on Binary Black Holes
NeuraChip
Network Attached FPGAs in the OCT
Monte Carlo eXtreme (MCX) – a Physically-Accurate Photon Simulator
Modeling Hydrogels and Elastomers
Modeling Breast Cancer Spread
Measuring Neutrino Mass
Investigating Mantle Flow Through Analyses of Earthquake Wave Propagation
Impact of Marine Heatwaves on Coral Diversity
IceCube: Hunting Neutrinos
Genome Forecasting
Global Consequences of Warming-Induced Arctic River Changes
Fuzzing the Linux Kernel
Exact Gravitational Lensing by Rotating Black Holes
Evolution of Viral Infectious Disease
Evaluating Health Benefits of Stricter US Air Quality Standards
Ephemeral Stream Water Contributions to US Drainage Networks
Energy Transport and Ultrafast Spectroscopy Lab
Electron Heating in Kinetic-Alfvén-Wave Turbulence
Discovering Evolution’s Master Switches
Dexterous Robotic Hands
Developing Advanced Materials for a Sustainable Energy Future
Detecting Protein Concentrations in Assays
Denser Environments Cultivate Larger Galaxies
Deciphering Alzheimer’s Disease
Dancing Frog Genomes
Cyber-Physical Communication Network Security
Avoiding Smash Hits
Analyzing the Gut Microbiome
Adaptive Deep Learning Systems Towards Edge Intelligence
Accelerating Rendering Power
ACAS X: A Family of Next-Generation Collision Avoidance Systems
Neurocognition at the Wu Tsai Institute, Yale
Computational Modeling of Biological Systems
Computational Molecular Ecology
Social Capital and Economic Mobility
All Research Projects

Collaborative projects

ALL Collaborative PROJECTS

Outreach & Education Projects

See ALL Scholarships
100 Bigelow Street, Holyoke, MA 01040