What are the benefits and risks of software engineering intelligence platforms?

Software engineering intelligence (SEI) platforms provide software engineering leaders and teams with insights into what engineering data is really saying about how software products are built and deployed and how software engineering teams are performing. They achieve this by collecting data from numerous engineering systems and employing sophisticated analysis to discover useful trends and patterns.

While there are clear benefits offered by SEI platforms, software engineering leaders must also be wary of the potential risks that may be incurred by using these platforms. Here are the key benefits and risks that software engineering leaders should consider when determining whether to invest in SEI platforms.

4 Benefits of SEI Platforms

The business insights generated from SEI platforms deliver several key benefits to software engineering organizations, including organizational effectiveness, velocity, quality and business value.

Organizational Effectiveness: SEI platforms measure and report an organization’s effectiveness by combining qualitative data, such as employee surveys, with quantitative data, such as time spent in meetings. This combination provides software engineering leaders with a more sophisticated view into the effectiveness of their teams. SEI platforms can offer advanced organizational effectiveness benchmarking capabilities. For example, software engineering leaders can compare their teams’ happiness scores with teams from organizations of a smaller size and in the same industry. This is something that could not be achieved with a homegrown solution.

Velocity/Flow: SEI platforms collect data from code control and versioning solutions, as well as from the tools used to deploy software. Collecting this data makes it possible to track key metrics around build and deployment velocity. SEI platforms increase the visibility of important flow metrics such as cycle time, deploy frequency, code review time and time to revolve pull requests. With increased visibility into velocity and developer productivity, software engineering teams and leaders are better able to demonstrate the business value of investments in developer experience.

Quality: SEI platforms use a variety of methods to measure and benchmark software quality. These platforms go beyond traditional lagging indicators, such as static code analysis, defect counts and code coverage, to also provide practice quality metrics. For example, SEI platforms use breakdown metrics that measure the percentage of code that is new versus code that has been refactored. Additionally, SEI platforms analyze patterns in code-versioning solutions and identify poor practices, such as long-lived pull requests or changes that make it to production either too quickly or without sufficient review.

Business Value: SEI platforms enable software engineering leaders to shift the focus from measuring output to measuring outcomes. These platforms do so by collecting data from Agile planning tools to help software engineering leaders demonstrate the business value of investments in developer experience by connecting engineering outputs to business artifacts. Furthermore, SEI platforms support the creation of strategic business objectives and key results linking these metrics to tactical key performance indicators in the software engineering domain.

4 Risks of SEI Platforms

Despite the number of benefits SEI platforms provide, software engineering leaders must be aware of the potential risks posed by these platforms, such as overlaps with existing markets, data security and perceptions of micromanagement.

Overlap with existing markets: The SEI platform market has some degree of overlap with the established value stream management platform (VSMP) market. VSMPs enable organizations to optimize end-to-end product delivery, but they take a broader view than SEI platforms. To mitigate this potential risk, software engineering leaders can use VSMPs to drive organizational outcomes by optimizing cost, operating models, technology and processes, while SEI platforms can be used to gain insights on the engineering part of the value stream.

Perception of micromanagement: Developers may feel that the collected data from SEI platforms will be used to spy on them. If developers feel this way, team productivity will suffer, as talented developers may leave the organization. SEI platforms can also be misused when software engineering leaders collect individual metrics instead of team-level metrics. If developers feel micromanaged or mistrusted, this will destroy morale. To avoid this, organizations should not use individual metrics to drive team decisions. Instead, rely on team measurements or value metrics. Partner with vendors who prioritize team and group level metrics over individual level metrics.

Data security: SEI platforms need access to sensitive and often proprietary data. Some organizations may not wish to give a third-party access to sensitive data. To avoid this, look for evidence that the vendor complies with the highest security standard, such as the Information Security Management System (ISMS).

Organization maturity: Not all organizations are at the maturity level where they will benefit from adopting an SEI platform. Software engineering leaders should carefully assess if an investment is warranted. Before deciding, organization should first pilot a software engineering insight program. This program should be a quick, low-cost experiment to collect a handful of key engineering metrics across the four benefits of SEI platforms. The program can be used to better understand any challenges with collecting data and to validate if the organisation is mature enough to change behaviors based on learning and insights from the experiment.

SEI platforms provide consistent visibility into team productivity and customer value that lead to successful business outcomes. Demonstrate the business value of software engineering teams by selecting SEI platforms that support linking of desired strategic business outcomes with measurable tactical engineering metrics.

Frank O’Connor is a Sr Director Analyst at Gartner, Inc. where he provides advice for software engineering leaders to tackle modern-day challenges and develop successful technology strategies. Gartner analysts will provide additional insights for software engineering leaders at Gartner Application Innovation & Business Solutions Summit, taking place May 22-24 in Las Vegas, NV.

Would you recommend this article?


Thanks for taking the time to let us know what you think of this article!
We'd love to hear your opinion about this or any other story you read in our publication.

Jim Love, Chief Content Officer, IT World Canada
Gartner, Inc. (NYSE: IT) delivers actionable, objective insight to executives and their teams. Our expert guidance and tools enable faster, smarter decisions and stronger performance on an organization’s mission critical priorities. To learn more, visit gartner.com.

Featured Download

IT World Canada in your inbox

Our experienced team of journalists and bloggers bring you engaging in-depth interviews, videos and content targeted to IT professionals and line-of-business executives.

Latest Blogs

Senior Contributor Spotlight