Four major Linux distributors have sharply criticized a recent report from Forrester Research Inc. that found that Microsoft Corp. outperformed them in responding to and fixing security flaws.
In a joint letter released April 6, Linux distributors Debian, MandrakeSoft Inc., Red Hat Inc. and SUSE Linux AG questioned the validity of Forrester’s conclusion and claimed that the report had “extremely limited real-world value” for users.
“It’s bogus in its current form,” said Joey Schulze, a member of Debian’s security team.
Laura Koetzle, author of the Forrester report, defended her company’s analysis of the data. All vendors studied in the report were measured equally using publicly available and widely accepted vulnerability rating measures from the National Institute of Standards and Technology (NIST), she said.
Cambridge, Mass.-based Forrester’s report, titled Is Linux More Secure than Windows? and released March 22, looks at how Microsoft and the four Linux vendors responded to reports of security flaws over a 12-month period, from June 1, 2002, to May 31, 2003. The document gives Microsoft the highest marks among the vendors for its “responsiveness” and its “thoroughness” in dealing with reported security vulnerabilities.
On average, Microsoft took 25 days after it became aware of a flaw to release a fix, and it was the only company to fix all vulnerabilities, the report stated. However, Microsoft also had the highest percentage of serious flaws.
In contrast, Moreno Valley, Calif.-based MandrakeSoft took 56 days on average to issue fixes for its Linux distribution, the Forrester report said. SUSE and Red Hat took 54 and 47 days, respectively, while Debian took an average of 32 days.
The Linux vendors also scored lower than Microsoft in terms of the percentage of reported flaws they fixed. Debian ranked last, fixing 275 out of 286 flaws.
While the data that the analysis is based on is accurate, the conclusions are not, said Vincent Danen, security update manager at MandrakeSoft. By treating vendor responses to all vulnerabilities as equal, the Forrester report failed to measure the much better record of Linux distributors when it comes to particularly serious flaws, Danen said.
Linux vendors typically treat flaws on a case-by-case basis, with high-risk flaws getting a higher priority than the low-risk ones, Danen said. The response to a flaw is based on risk assessments made by each distributor and may not always coincide with the assessment made by a third party such as NIST, he said.
“Our users will know that for critical flaws, we can respond within hours,” SUSE Linux said in a statement. SUSE is now owned by Novell Inc.
By focusing purely on quantitative analyses, the Forrester report fails “to differentiate between both the seriousness of the flaws and, more importantly, the quality of the fixes,” SUSE said in the statement.
Making a distinction between serious flaws and the not-so-serious ones when devising a response is crucial, said Joe Poole, manager of technical support at Boscov’s Department Stores LLC in Reading, Pa.
“(Vendors) have to separate the things they need to do tomorrow and what can be done in a month,” he said.
This isn’t the first time that a Forrester report comparing Microsoft products and Linux has been criticized. In September, the research firm drew flak from users over a report that showed Microsoft development platforms having a substantial cost advantage over Linux/J2EE for portal applications, since that report was actually funded by Microsoft. Koetzle stressed that her security report had no Microsoft backing whatsoever. “Microsoft did not pay for this report,” she said.
