The new server is equipped with an APC Smart-UPS 750VA LCD 230V APC SMT750I for protection against power failures and electrical surges. This is my first new UPS, I had some old huge ass UPS but that thing was end of life when I started using it… which I shouldn’t have. This UPS is brand new with a battery that is still in optimal condition.
The display is handy and gives more accurate information about the UPS, Hardware and Source Utility then the previous LED only Smart-UPS series from APC. It can cycle messages about source utility voltage and frequency, UPS load in both Watt and VA, battery run-time and battery load percentage. It can save you a login into the PowerChute software or UPS Agent website. In combination with the menu control buttons you have more functionality available directly on the UPS. There are a couple of LEDs still present for the important system status functions. Good decision from APC, since the LEDs give a quicker insight of status and issues then the display.
Window Server 2012 R2 detects the UPS out of the box as a UPS HID device. The UPS behaves like a battery in a Laptop. You can configure actions in the Configuration Power section that the system should shutdown if the battery drops below a certain percentage, but it is too limited. That’s why I was planning on using the APC UPS Daemon…
APC UPS Daemon
The SMT750I a UPS of the new generation from APC which utilizes USB. I regret that a little bit because the new generation is not compatible with APC UPS Daemon, as stated on their website. This means that you must use the APC PowerChute software.
APC PowerChute Business Edition
The idea and architectural concept of the PowerChute software is good, run a PowerChute Agent on every physical server that has a UPS connected. Install one PowerChute server that links the agents together. Use a separate PowerChute console application to manage it all. Additionally, the PowerChute agent provides a secure website to manage a UPS individually and SNMP support. This all is free as long as you only need to link up to 5 agents together. Need more, you have to pay for the 25 units version.
PowerChute Java Runtime Environment
The real-world scenario is that PowerChute requires the Java Runtime Environment, which has been troubled a lot by slow security patching since Oracle is maintaining it. Not speaking about the performance and memory consumption. All aspects that are important in a server environment where you want to keep the attack surface low and performance optimal. It is a major disappointment that the PowerChute installer is delivered with JRE 7 Update 25! That version was released on April 16th this year, more than 7 months ago! Worse is that Update 45 released on October 15th this year contains 51 security fixes. So installing PowerChute Business Edition 9.1.1 also installs a total number of 51 known security vulnerabilities on your system. Great Job APC!
You can update the Java Runtime Environment of course, but APC states in their PowerChute Business Edition Product Compatibility Chart that the software has been verified with JRE 7 update 25. Running the PowerChute software with a newer version is more or less your own risk. Just like the 51 known security vulnerabilities and counting…
Update Java Runtime Environment for PowerChute
It would make sense to install the latest Java Runtime Environment from Oracle in advance of installing any of the PowerChute software components. Assuming that the PowerChute installer doesn’t install its bundled JRE when it detects a newer JRE version.
Like we’re all thought in life, never ASSUME! (Or you make an ASS out of U and ME) So that assumption is wrong, PowerChute still installs its own bundle. It is actually not a real install, it is more like bundling the JRE files with the installer. You will find them under:
C:\Program Files (x86)\APC\PowerChute Business Edition\jre1.7.0_25
It gets even worse, it will always use the bundled JRE version unless the software is configured differently. Something that needs to be redone after each update of the PowerChute software or update of the JRE. APC has a special tool for this called PowerChute Business Edition Java Runtime Environment Config Tool, or short as PcbeJreConfigTool. I tried to download it from their site, but the FTP link was broken at that time. Found a mirror on their Japanese FTP Server: PcbeJreConfigTool.exe Here you can find the manual of the PCBE JRE Config Tool.
There are various configuration options which are easily configured through the PowerChute Agent Website.
There are a lot of events that can be configured to trigger e-mail, shutdown or a custom script.
For outgoing e-mail are of course the well known SMTP settings present.
There are 4 strategies for when the power fails. Depending on the scenario you want to directly shutdown or wait a little while. I chose that option with a shutdown delay of 120 seconds. This is all running at home and the 120 seconds might give me time to restore power without the need of a complete shutdown of the server. After that I give the Virtual Machines 10 minutes to shutdown and then the OS 180 seconds. After that the UPS will terminate the outlets. This strategy assumes that the UPS will have enough battery runtime.
Another option is to use the remaining runtime as trigger to start the shutdown process.
If generators are around, then you just let it run until the generator is up and running to take over the utility power.
The shutdown configuration is better visible at the Outlet Sequence page.
It is there, not much more to say about.
Event and Data Log
The Event and Data Logs are gathered and maintained by the Agent itself. Thus when the agent is not running, no information is logged.
It is possible to configure the logging to keep all logs like I did, so I have some nice historical data over time.
The new generation of UPS models from APC has a green mode that saves energy by bypassing certain circuits that are not needed if the utility input is within normal parameters, as they state in their FAQ. I understand the feature and it makes sense, however I doubt their statement that they still provide the same protection as when not using the green mode.
When terminating the utility power, a relay click is audible. What really happens in unclear to me and I don’t have the right equipment to measure the output of the UPS. If someone can provide more insight or reference test results, I would love to hear about it.
I selected the UPS based on the fact that it should give enough uptime to shutdown all the Virtual Machines and Host OS. The server load on the UPS is 10-15% when idle. Note that not all hard disks are present yet, so this might increase slightly in the near future. The low load results in a battery runtime of >90 minutes, which is long enough to shutdown the system including the virtual machines and leave margin for battery degradation over time.
Shutdown == Virtual Machine Termination
To shut down my Virtual Machines I configured them all to shut down when the Host OS shuts down. This doesn’t give me control of the order in which the shutdown occurs, and results in a high disk load since all machines shutdown at the same time. But then again, this is only in emergency situations which shouldn’t occur that often or better even never.
I can also save the Virtual Machines, but in the past this was not recommended/supported for Virtual Machines that are domain controller. Which a few of them are and I’m not sure whether this has changed for Windows Server 2012R2.
As good practice, testing these scenarios is important because practice makes perfect. So having all the virtual machines up and running, after making a back-up of course, I unplug the power cord of the UPS.
The UPS display counts down 120 seconds until shutdown is initiated. Then the Windows host OS shuts down in 5 seconds and the computer is turned off. The UPS display counts down again, before it disables its power outlets.
The 5 seconds shutdown of the host OS has made me already skeptical, I find it hard to believe that 8 virtual machines shutdown that quick.
After restoring power the UPS counts 60 seconds before it enables its outlets again. The server is configured to always switch on after AC Restore and starts booting up. My virtual machines are configured with an always start delay and start running again soon after the server has booted. Logging into the first Virtual Machine I get a prompt why the server was not cleanly shutdown…
It looks like the shutdown initiated by the PowerChute agent forcefully shuts down the server and doesn’t wait for the Virtual Machines to complete their shutdown routine. Thankfully I made a backup of the Virtual Machines.
So I’ve added a PowerShell Virtual Machine Shutdown Script that shuts down the Virtual Machines and now also takes care of the order in which the shutdown occurs as well as spreading the shutdown load a little over time.
Install only the PowerChute Agent, which provides you with all the options you need. The Console and Server only add basic reporting over multiple UPS units.
Install the latest version of the Java Runtime Environment and use the PcbeJreConfigTool to switch to the new Javan Runtime Environment.
Write a script to safely shutdown your Virtual Machines, or Business Applications to prevent data loss.
The hardware is great, APC builds quality hardware products that perform correct if you maintain them well and replace the battery in-time.
The PowerChute Software works, but you need to put some serious effort into it to get it properly up and running. APC should be ashamed for delivering an out-dated JRE with 51 known security vulnerabilities. The PowerChute agent works OK and looks decent, but the shutdown procedure does not really help protecting your data and/or business applications without proper scripting. The Console and Server are not worth the installation.