September 2001
Volume 3 • Number 4

ContentsSo

STANDARDS SYMPOSIUM
Help, Hindrance, or Delusion?

I invited members of SQP’s Editorial Board to comment on the development and use of technical standards in software and quality engineering. I knew there were certainly open questions about existing and emerging consensus-based standards. I also knew these individuals had plenty of firsthand experience with these issues.

Here are some of their e-mail exchanges.

– Taz Daughtrey

John Horch: “As you know, I’m heavily into software standards development, as are other editorial board members. While we strive to make the standards palatable, useful, and realistic, there seems to be little hard data reflecting our success. Reflecting on our own experiences, published experiences, hearsay, apocryphal evidence, or just wishful thinking, we might uncover some usage information, interest others in participating, discover that the fruits of our labors are worthless, or find that we are giving a party and nobody cares.”

Mark Paulk: “In talking about these standards, I think it may be helpful to distinguish between ‘process’ standards and ‘product’ or engineering standards. The importance of product standards is well known. Without a standard gauge, railroad cars could not move from one country to another—or one rail company to another’s lines—because of different widths and sizes of rails. Without standards for connecting telephones, computer networks, peripheral devices, and so on, modern industry would collapse. Note, however, that the bulk of these standards deal with hardware, and most of the standards that are universally accepted as being valuable are interoperability kinds of standards.”

Denis Meredith: “One difficulty with process standards—to add to Mark Paulk’s thread—
is determining whether they have been followed. If the process produces a product or products, and those products conform to their product standards, is that evidence that the process
standard has been followed? If not, what would be evidence?

“Like some others on the editorial board, I have worked with working groups for both ANSI/IEEE and international standards. The professional growth from that experience aside, I find that the most valuable aspect of standards for me is in working with clients. When I work with a client who does not already have a document template, for example, I go to the appropriate standard for a starting point. I also borrow liberally from process standards when the need arises. From a productivity standpoint, I don’t have to invent something new each time, I am unlikely to forget something important, and if we decide to do something different from the standard it is done deliberately and not from ignorance.”

Deependra Moitra: “There is no question whether standards are useful or not, but the question is whether the existing standards are yielding the expected results, and whether new standards should be developed for the ‘new’ way of doing software engineering. It must also be debated whether an IEEE-like generic standard is what we need—especially for processes—or whether development and application of standards should best be left to consortia of companies to suit their business context. While I do agree that we need product, programming, and ‘interface’ standards, I am of the opinion that we don’t need process standards.

“My expectation is that the standards community needs to keep pace with changing times and product development practices. For example, do the existing standards apply to the e-commerce and m-commerce systems development, and even if they do, how well do they? I would say that the existing standards, especially software engineering and quality standards, require a relook for the current-day, highly volatile, fast-paced development.

“Also, it needs to be evaluated whether claims of compliance to standards are being used just as marketing instruments. I suspect this is largely the case with ISO 9001 or the Software Capability Maturity Model (CMM). For example, I have seen a lot of supposedly Software CMM level 5 companies where nothing has actually changed for the recipients, the customers. I believe we need standards for products, components, and ‘interfaces’ but not for general software development process lifecycle. I say this because in hardware I have found standards to be helpful as I could objectively certify the compliance to the standards in a given situation. But in software, certifying compliance is very subjective.”

Mark Paulk: “I’ve been involved with a number of different process models and standards. I have seen some excellent work resulting from the use of those models and standards to improve the way companies do business. I have also seen abuses—to the point that I would sometimes characterize them as active stupidity or even unethical behavior. In his book, Measuring and Managing Performance in Organizations, Rob Austin makes the point that when we use measurement for motivational purposes, we will cause dysfunctional behavior unless we are adequately measuring all the important characteristics of the system of concern. The importance of ISO 9001 certification or achieving level 3 has caused many organizations to focus on a label rather than the intent behind our models and standards.

“The organizations that are using process models and standards effectively are the ones that keep their business objectives firmly in view and shape their improvement (or conformance) programs to achieve improvements in productivity, quality, customer satisfaction, and so on. The ones going for a certificate frequently don’t make the necessary behavioral changes to affect the bottom line.

“This is something I have struggled with for years. Software capability evaluations have incentivized the rapid adoption of the Software CMM...while at the same time inspiring behavior that has caused much of the criticism of the model. Similar comments can be made about ISO 9001 and pretty much any of the other process models and standards that I am aware of. Customers have a legitimate need to select and monitor qualified suppliers...and process capability is a critical component of being qualified to do work. General-purpose models and standards can be extremely useful to customers in assessing their suppliers and provide a common framework for customer-supplier discussions.

“I don’t have a solution for this problem. The dilemma is one that we need to be sensitive to in both writing standards and using them.”

Here is where we invite you to join in the discussion. What do you think about the points raised previously? What have been your experiences in applying software quality standards? What new directions should the professional community be trying?

For example, do you believe the customer focus and process orientation of the revised ISO 9001 standard make it more applicable and useful for software developers? Are process improvement and business excellence themes being sufficiently captured in newly developed standards?

Please contact me at sqpeditor@aol.com with your comments. With your permission, some of these responses may be published in subsequent issues of this journal. I would be especially interested in submissions (either short reports or full-length articles) reporting on your experiences with standards.