DevOps is maturing, but teams are plagued by alert fatigue and still lack information to solve problems

Alert fatigue got worse for IT staff on-call in 2015, according to this year’s State of On-call report from VictorOps Inc.

The annual survey of more than 600 respondents found that alert fatigue is the number one pain point, with 70 per cent saying it was an issue in their organization – up 10 per cent from last year. “As infrastructures get more and more complicated, and more monitoring systems are tied in, alert fatigue and noise in the system gets to be a big deal,” said VictorOps CEO Todd Vernon. “It’s not surprising. It’s a tough job to do.”

Alert fatigue is the phenomenon of constantly getting paged for non-actionable alerts, including false alarms. Other top pain points, according to the survey results, include on-call staff not being sure what’s happening across all systems when they get alerted, an organization’s leaders not having context around what’s happening during a firefight, and not having the relevant information to solve a problem on their own.

Not surprisingly, Vernon added, automation tool use is continuing to grow as infrastructures continue to become more complex, and those tools are becoming more robust; seventy per cent reported regular use of automation tools such as Puppet and Chef, up from 58 per cent last year. “The most popular practices cited by people in order of importance are automation and culture.”

However, email remains the number one channel people learn about problems, said Vernon. “That is unfortunate, because it is probably also the worse way to learn about problems.”

He said the use of status pages to communicate to more stakeholders during a firefight is also growing, as is the use of ChatOps, with 40 per cent of respondents using it in 2015 compared to 28 per cent last year.

The majority of the respondents to the survey were from “Internet forward” companies, including software, Internet service, media and entertainment companies, and were an on-call team member or managing a team. Their roles included system admins, developers and programmers.

The annual State of On-call survey was started two years ago to better understand how agile development and the pressure of continuous deployment was putting on technical operations teams of all sizes, said Vernon. “What used to be people bolting boxes onto racks is now engineers writing code.” More and more organizations are putting developers on the front lines to enable rapid response and resolution of technical problems, he said. “The clear signal over the last year has been the death of operations as the lone protector of highly technical infrastructures and systems.”

Vernon said the survey shows that DevOps is maturing, with the majority of respondents saying they consider themselves at least minimally knowledgeable about DevOps, with 76 per cent reporting now reporting they have a year or more experience in DevOps as compared to 52 per cent last year. “DevOps is a real thing out there and we see it more and more every day.”

Would you recommend this article?


Thanks for taking the time to let us know what you think of this article!
We'd love to hear your opinion about this or any other story you read in our publication.

Jim Love, Chief Content Officer, IT World Canada

Featured Download

Gary Hilson
Gary Hilson
Gary Hilson is a Toronto-based freelance writer who has written thousands of words for print and pixel in publications across North America. His areas of interest and expertise include software, enterprise and networking technology, memory systems, green energy, sustainable transportation, and research and education. His articles have been published by EE Times, SolarEnergy.Net, Network Computing, InformationWeek, Computing Canada, Computer Dealer News, Toronto Business Times and the Ottawa Citizen, among others.

Featured Articles

Cybersecurity in 2024: Priorities and challenges for Canadian organizations 

By Derek Manky As predictions for 2024 point to the continued expansion...

Survey shows generative AI is a top priority for Canadian corporate leaders.

Leaders are devoting significant budget to generative AI for 2024 Canadian corporate...

Related Tech News

Tech Jobs

Our experienced team of journalists and bloggers bring you engaging in-depth interviews, videos and content targeted to IT professionals and line-of-business executives.

Tech Companies Hiring Right Now