From Shoes to Software, Benchmarking Helps Organizations Find the “Right Fit”

By John Ohab

This blog post was shared with us by the Data & Analysis Center for Software (DACS). It is the 15th entry in our 22-part series produced by the Defense Technical Information Center (DTIC).

In the early 80s, Xerox began experiencing fierce competition from both U.S. and Japanese competitors. New entrants from overseas were rapidly gaining market share. David Kearns, then CEO, stepped up and began a corporate push for lower manufacturing cost, and increased quality control. His program, known as “Leadership Through Quality” not only helped turn Xerox around, but it also gave birth to what we know as modern-day “benchmarking.”

But what exactly is benchmarking?

Benchmarking is a process for finding the world-class standards for a product, service, or system and then adjusting one’s own products, services, and systems to exceed those standards. When properly applied, benchmarking can force an organization to take a hard look at its own performance compared to that of its peers. It forces an organization to look at best practices within their particular industry and determine whether there is room for internal organizational improvement.

Today, we often hear the term benchmarking in the information technology and software industry, but oddly enough, its roots can be traced to the shoemaking industry. In the nineteenth century, cobblers would measure a client’s feet for handmade shoes. The cobbler would place a client’s foot on a “bench” and “mark” it out to make the pattern for the shoes. This pattern became a reference point for the cobbler and helped ensure a better fit. From benchmarking shoes to benchmarking software, this process has helped improve countless organizations.

For more than 15 years, the Data & Analysis Center for Software (DACS) has worked to benchmark the performance results of best practices used by the software industry. This is important for two specific reasons. First, there is a significant gap in the knowledge base surrounding best practices performance in the software industry. Second, many organizations struggle to justify the investment cost required for popular development methods such as Capability Maturity Model (CMM) and Agile Development.

So, what became of 15 years of data collection and analysis? As it turns out, many of these popular methods have shown significant improvements in product quality (as much as a 2-times improvement), productivity (50-75% improvement is typical), and with significant return on investment (many organizations reporting 3 to 1 or better). Not only did quality, productivity and ROI improve, but organizations also experienced a number of secondary benefits. Organizations observed that they had: lower employee turnover, improved employee morale, improved customer satisfaction, less overtime required by the employees, and more repeat customer business.

A Business Case for Software Process Improvement (2007 Update): Measuring Return on investment from Software Engineering

In the spirit of benchmarking and continuous improvement, the DACS has begun a brand new data collection effort to support Government project cost estimation and project management benchmarking efforts. They are developing a repository of software and systems engineering projects, which will include cost, schedule, and other project planning data (such as risk information, lessons learned, project templates, and estimation heuristics). The DACS expects that these new benchmarks will be used by project managers and by researchers in the development of new models for further organizational improvement.

The Data and Analysis Center for Software (DACS) serves as the authoritative DoD source for state-of-the-art software engineering and technology information to support the software community. It provides a centralized hub for collecting and distributing software data and information and offers technical support for acquiring, developing, testing, validating, and transitioning software technology and processes.

Interested in learning more or working with DACS on an upcoming effort? The DACS can be reached via the IAC website or the DACS website. While at the DACS website, check out its many resources, including quarterly technical journals (Software Tech News), online web-based courses, technical reports, best practices reports (called Gold Practices), and ROI data from process improvement (ROI Dashboard).