Ted Maulucci got the good news and the bad news on the same day.
The CIO of Tridel Corp., one of the largest builders of residences and condominiums in the Toronto area, he’d largely been in charge of software development for the company’s 15 divisions, which range from a law firm to property management.
But in 2005 responsibility for the data centre was put into his hands.
It was not in good shape for a system running about 80 virtual servers for some 500 desktops spread across 20 locations on shared services model.
The Canadian Oxford dictionary defines chaos as “utter confusion.” That was pretty well what the affable Maulucci faced.
“I was left with a mess,” he recalls. “Just about everything imaginable was wrong with what I was given. I still had machines running Windows 95. The file servers were 90 per cent full. We had no fire suppression in our server room. One thing or another had to be upgraded.” On top of that, the nine-year Cisco core switch was past its expected lifespan and had to be upgraded.
Core switches aren’t called core for nothing, but in this case other things - like the 18-year-old air conditioner -- needed priority. Besides, switches are solid state, right? Management was told about the situation and approved his priorities. He gambled on the switch.
“We ended up winning the lottery,” he laughs, “and it did fail.” And on the day consultants were scheduled to drop in and chat about its replacement.
This isn’t a Cisco-bashing story - actually, Maulucci is very complimentary to the company for rushing to find replacement board -- but about what managers should do when suddenly faced with a chaotic situation.
Darin Stahl has faced more than a few of these. Now research stream lead for data centres at London-Ont., based Info-Tech Research, he’s been hired by a number of companies as a manager or consultant to clean up other people’s messes.
On one memorable job interview, he was asked one question: “Have you ever fired anyone?” There’s a warning signal.
“That particular place was pretty horrible,” Stahl says. It was a financial services company with a “dog’s breakfast” of platforms on outsourced mainframes and client-server systems which delivered information to brokers around the world. However, it was disorganized to the point where the network administrator “was a help desk guy three of four weeks ago.” Small wonder competitors were delivering better IT service and morale was bad. By the time Stahl got there “everybody else had been fired or never came back from lunch.”
Still, that topped one company he was called in to rescue where money was so tight IT staffers were building their own routers from parts bought on e-Bay.
Thrown into situations like this, it’s not enough to say you’ve got to prioritize, says Stahl, nor can you automatically spend on hardware to get your way out of it. A manager has to look at an IT department in cases like these as if it was being evaluated for a corporate acquisition in three categories:
People: Go through the staff files, looking at their performance evaluations, what challenges they’ve faced and what they’re working on now. Talk to them, asking about their goals. Then lay it on the line in a firm but encouraging way by saying in one way or another, “I’m here to be your best friend or worst enemy -- and hopefully your friend. We’re going to turn this around. Let’s find some common ground so we work together for the shareholders, the company and our careers.”
Processes: How does the IT department work? Are there change control documentations? Do they do the same things the same way as a form of quality control? Look for internal or external audits, which will show warning signs. Talk to key internal and external business users to find out what they’d like to see and to understand the IT situation from their perspective as well as the company’s business processes.
Technology: It’s not merely about getting an inventory of what’s there. If there are any, freeze the work of consultants – you don’t know if you need them. See if they’re adding value before renewing or extending their contracts. Better than a hardware inventory is doing a zero-based budget: Start with a clean sheet of paper and build every line item from scratch, looking at every piece of hardware, every licence term every piece of maintenance. That, Stahl says, will force you to question everything and lead to discoveries of systems that were supposed to be shut down years before but are still running.
By doing these three things together you can establish priorities, he says. In the case of the financial services company, he learned that the real customers weren’t the internal staff but the brokers buying its monitors and data. To resolve service problems, he worked with an outside software developer to Web-enable all of the back-end functions through a portal.
There may be a sense of obligation to run to the CEO and spill all. But Jason Bremner, director of infrastructure hardware at IDC Canada, says management needs to know about risk. So he advises getting a handle on where the problems are and what needs to be done before things break down. Determine what IT systems are at the greatest risk to the business. In turn, find where changes can be made that will give the greatest gain that affect the organization the least. If necessary, that will help create a plan to take to management, whose input will guide you in making priorities.
In Maulucci’s case, management knew about the risk to the switch and that other work had priority. So when it failed, he wasn’t raked over the coals.
On the other hand, he adds, it did give managers an awareness of the importance of IT, which helped get additional funding.
As for replacing the switch, Maulucci when he was researching possibilities he ran into a friend at a golf tournament sponsored by a vendor who recommended he look at equipment from another manufacturer. When the switch blew he decided to buy it and has been satisfied.
Seems those golf tourneys are valuable after all.