Alert fatigue got worse for IT staff on-call in 2015, according to this year’s State of On-call report from VictorOps Inc.
The annual survey of more than 600 respondents found that alert fatigue is the number one pain point, with 70 per cent saying it was an issue in their organization – up 10 per cent from last year. “As infrastructures get more and more complicated, and more monitoring systems are tied in, alert fatigue and noise in the system gets to be a big deal,” said VictorOps CEO Todd Vernon. “It’s not surprising. It’s a tough job to do.”
Alert fatigue is the phenomenon of constantly getting paged for non-actionable alerts, including false alarms. Other top pain points, according to the survey results, include on-call staff not being sure what’s happening across all systems when they get alerted, an organization’s leaders not having context around what’s happening during a firefight, and not having the relevant information to solve a problem on their own.
Not surprisingly, Vernon added, automation tool use is continuing to grow as infrastructures continue to become more complex, and those tools are becoming more robust; seventy per cent reported regular use of automation tools such as Puppet and Chef, up from 58 per cent last year. “The most popular practices cited by people in order of importance are automation and culture.”
However, email remains the number one channel people learn about problems, said Vernon. “That is unfortunate, because it is probably also the worse way to learn about problems.”
He said the use of status pages to communicate to more stakeholders during a firefight is also growing, as is the use of ChatOps, with 40 per cent of respondents using it in 2015 compared to 28 per cent last year.
The majority of the respondents to the survey were from “Internet forward” companies, including software, Internet service, media and entertainment companies, and were an on-call team member or managing a team. Their roles included system admins, developers and programmers.
The annual State of On-call survey was started two years ago to better understand how agile development and the pressure of continuous deployment was putting on technical operations teams of all sizes, said Vernon. “What used to be people bolting boxes onto racks is now engineers writing code.” More and more organizations are putting developers on the front lines to enable rapid response and resolution of technical problems, he said. “The clear signal over the last year has been the death of operations as the lone protector of highly technical infrastructures and systems.”
Vernon said the survey shows that DevOps is maturing, with the majority of respondents saying they consider themselves at least minimally knowledgeable about DevOps, with 76 per cent reporting now reporting they have a year or more experience in DevOps as compared to 52 per cent last year. “DevOps is a real thing out there and we see it more and more every day.”