Commissioning of mission-critical data center systems is essential to ensure highest system availability through different phases of operation – at the outset of operations, down the road at full system buildout, and, later, when systems begin to age.
One challenge to successful data center commissioning is that new data centers are often built with excess HVAC/electrical capacity to allow for future system expansion. In this case, both thermal and electrical design loads may not be achieved for months or years from the beginning of operation. This leads to two different sets of problems at different phases of the system lifecycle.
Figure1. A Data Center, Photo Credit: Wikimedia Commons
Insufficient System Turndown
When first operational, if a low percentage of the design data center system capacity is built, HVAC systems may have insufficient turndown to operate well at low loads. Turndown refers to the ability of a piece of equipment or system to operate at part-load conditions. While some modern equipment has the ability to operate at a high turndown ratio, it is typically not infinite. In other words, at some point, the controls are still flipping an oversized piece of equipment (e.g. a chiller) on and off to meet the setpoint. This cycle can result in potential space/rack subcooling when the equipment is on, and then difficulty meeting setpoint when the equipment is off. If these cycles are short, the phenomenon can result in excess equipment wear and premature failure. To mitigate against these issues, the controls contractor and/or equipment manufacturer’s representatives may tune the systems/equipment to operate smoothly at low load; however, as loads increase over time, these initial configurations may lead to unexpected challenges and require readjustment, often well after the warranty period has expired. Testing at these low load conditions is an important part of data center commissioning, however, the focus is often on testing for design maximum.
Delay in Achieving Design Load
On the opposite end, as compute loads and rack buildout continue to increase to design levels, systems which may not have been tested at full capacity may begin to reveal issues that, while having been present the whole time, are now only making an impact as the loads approach design maximums. These issues could include chilled water flow/balance issues for water cooled equipment, hotspots on airside distribution, circuit breaker ampacity/settings/configuration errors, or even electrical termination issues not identified under part load conditions. When these issues result in failures, it is often the most damaging/expensive to an organization since, at design loads, server system utilization is also typically very high. For this reason, testing to design load conditions is essential as part of the commissioning process.
Commissioning for Full Range of Operation
In order to successfully test thermal management and power systems before server and network gear is online, it is essential to have a flexible platform for simulating – as accurately as possible – a variety of thermal and electrical loading scenarios, both initial loading and design buildout. Often times, load testing has been accomplished by utilizing larger load banks placed either in an open air server room even before rack gear is installed, or by using several load banks placed in the hot aisle(s). The latter approach is more appropriate for modern data centers utilizing hot aisle containment architecture, but still does not accurately model actual rack loads and can result in a failure to identify hot spots and other performance anomalies.
Benefit of Rack-Mounted Load Banks
The most effective approach to power and thermal load testing during commissioning that we have found is to utilize rack-mounted load banks (Figure 2). These are resistive heating element load bank units that mount in typical 19” server racks, plug into dual-circuit rack PDUs (power distribution units), and can be dialed-in to the expected load of the particular rack. In this manner, the full load rack configuration can be simulated, as well as some partial theoretical/planned initial buildout. The fact that the loads can be configured to match expected loads, and at specific locations, can have a number of benefits to more representative testing. For example, hot aisle hot spots can be better detected, resulting from either airflow variations within the hot aisle or, perhaps, based on variations in performance by in-row cooling hardware. Further, N+1 testing of cooling equipment can be accurately simulated. If, for example, a data center is designed to to have 10 in-row cooling (IRC) units, but only requires 9 operational to meet heat rejection loads (i.e. 9 + 1 = 10), during design load testing, a single IRC can be failed and the hot aisle can be evaluated to determine if unacceptable hot aisle temps result near the failed IRC unit. Performing this kind of testing with floor load banks would be inconclusive at best.
Figure 2. Simplex MicroStar Precision Air Heater (Rack-Mountable Load Bank), Photo Credit: Ben Fowler
Benefits Come at Added Cost
While rack mounted load banks are the most accurate way of simulating a variety of loading conditions during data center commissioning, they are more expensive than a few larger load banks. The expense comes in the cost of the many individual rack mounted units, which, for the same connected kW, are more expensive than a few larger load banks. However, labor costs of rack mounted load banks are also more expensive. Mounting, installing blank-off panels, configuring, operating and removing 1-2 load banks per rack can be time consuming. When compared with the costs of system downtime and out-of-warranty service calls, the marginal costs for most organizations may be easily justified. As always, this is best evaluated for each particular project/Owner. The most important step, regardless of the form-factor of load bank testing, is that someone – whether a building Owner’s representative or a 3rd party commissioning authority – verifies system installation and functional performance before mission critical systems goes into service in order to achieve highest system up-time.
To learn more about data center testing and operation, check out some of Cx Associates’ other blog posts:
- “Data Center Efficiency: IT Side Approaches” by Ben Fowler,
- ”Direct Liquid Cooling for Data Center Energy Savings” by Brent Weigel,
- “Rethinking Server Room and Small Data Center Configurations” by Eveline Killian, and
- “Testing an Emergency Power System for a Data Center” by Katie Mason.
