There’s an old British saying that goes ‘penny wise and pound foolish.’
A southern Ontario hospital found that out the hard way three years ago when it decided not to follow an IT staffer’s recommendation and spend about $8,000 replacing the batteries in its universal power supply, which gave backup power to its servers.
That cost Brant Community Healthcare System some $40,000 to buy new servers, plus untold costs in labour, when the power failed and knocked out its IT system for three days.
The incident led to the institution getting a whole new virtualized server infrastructure.
A year later CIO Tim Armstrong looks back on the experience with some relief, but at the time it was a crisis. In fact it occurred during his first week on the job.
Armstrong had been hired by BCHS in 2011 after spending several years in IT positions working for Tier 1 auto parts manufacturers.
BCHS includes the Brantford GeneralHospital, a 265-bed acute car hospital, and the WilletHospital in nearby Paris. Combined there’s a medical and administrative staff of 1,500.
An IT staff of about 15 with an annual IT budget of $2.8 million looks after more than 100 servers, many of which were around a decade old. Downtime was not uncommon.
Several days after Armstrong arrived there was what he calls a “blip” in the power. “The servers,” he recalled, “went down very hard.”
That’s when he learned the real situation.
Like all hospitals in Ontario, BCHS’ IT budget had gone under the knife over several years. As a result, the department’s policy was to replace servers when they died. And that wasn’t uncommon because the servers didn’t have redundant power supplies or network interface cards.
Nor were there spare servers sitting around, so when one went down it took a few days to replace.
“We were averaging 18 down days a year related to unplanned hardware failure,” Armstrong says.
And if the server that was knocked out was one of the several needed to run the Meditech electronic health records system, it would take medical staff several extra days to catch up.
Coming from the private sector to a publicly-funded hospital, Armstrong had expected better. “I was absolutely shocked at the amount of poorly planned infrastructure for maintaining a medical facility … I was like they were stuck back in the 1990s.”
Not only that, the cost of maintaining the infrastructure was high.
After delving into the system Armstrong says executives and the board of directors were “somewhat surprised” with what he reported.
IT hadn’t communicated to management the risks of the way things were being done. Nor did the board understand the savings of taking advantage of leasing and virtualization.
“It was always seen that IT could be cut before some other areas.”
In the end — with several days of downtime and a $40,000 bill — it wasn’t hard to persuade the board that an RFP to completely replace the existing servers, storage (from the existing direct to a SAN) and backup was needed.
Armstrong was able to show the board that by leasing a new system maintenance costs could be cut enough to afford to pay for new clinical applications.
A key requirement was management and monitoring tools, so the IT department would have visibility into the infrastructure and have some warning if, as Armstrong put it, “something was going sour.”
The 75-page RFP, issued in the fall of 2011, included technical requirements for redundancy, high-availability and support for Meditech,
In fact, virtualizing the complex Meditech software was one of Armstrong’s worries.
“Just before we started another hospital in our LHIN (local health integration network) had abandoned their migration to VM(ware) and had gone back to physical servers, so I was quite paranoid that would happen to us.”
Some 15 vendors, including IBM, Dell, Cisco Systems Inc. Hewlett-Packard Co. and their partners replied to the RFP. The winner, HP [NYSE: HPQ], was selected for a combination of meeting technical requirements plus price — less than $1 million spread over five years.
The hardware was bought through Softchoice, while a Cleveland Park Place Technologies data centre services provider help with the Meditech migration.
But Armstrong credits HP’s expertise with aiding in getting the E.H.R. to run in a VMware environment.
The implementation of the new system was done between April and October, 2012, and is built around a single HP blade chassis with a number of blades, 160 physical and virtual servers, an HP EVA storage area network with 45 TB of capacity, and a backup disk and tape system.
A test environment was created to validate the methodology for moving to the new system, which took eight hours.
Not surprisingly with a new system, Armstrong reports there has been zero downtime in the 12 months since and application response time is up.
And the board has learned that cutting IT has a price.