September 2000
Volume 2 • Number 4

Contents

Manuscript Evaluation Criteria

Submissions to Software Quality Professional are sent for evaluation to members of the journal’s Editorial Board and to volunteers on its review panel. These evaluations are done in the categories shown below by identifying characteristics exhibited by the manuscript in each category.

Explicit criteria are provided for scores of 0, 2, and 4. For each evaluation item, reviewers assign a score based on the example that seems, in their opinion, to best fit the manuscript being evaluated. Reviewers use scores of 1 and 3 as intermediate values, when needed, to bridge the descriptions between scores.

1. How well do the authors identify the significance of the article’s content?

   0      The manuscript fails to identify an explicit significance for the content and offers no rationale for a reasonable reader to identify such significance implicitly.

   2      The manuscript makes a weak argument for the significance of the content, or the argument is convincing but expressed poorly. Many readers will not be convinced, or will miss the point.

   4      The authors explicitly identify the problems addressed in their manuscript and discuss the implications and costs associated with those problems. The authors also help readers understand the problem beyond the local context in which it is presented so that a discussion of a local specific problem and solution becomes applicable across a broad range of settings and organizations.

2. How timely is the article’s content?

0      The manuscript addresses a theme or subject that is obsolete or has been so overdone as to be considered effectively obsolete within the profession.

2      The authors address one or more issues that are discussed or addressed in the trade literature on a regular basis and are of continuing interest and are likely to be of concern to readers.

4      The authors identify a problem that will have impact in the near future and has not been previously addressed, or they explicitly identify external issues in recent or forthcoming industry current events using a theme or subject area that experts would consider timeless.

3. How well does the article apply accepted quality software principles to practical situations?

0      The use of quality software principles is inappropriate to the subject matter or is incidental with no direct bearing.

2      The article makes little or no use of quality techniques and principles, and offers no explicit rationale for such omission.

4      The manuscript builds its discussion or argument using a series of commonly accepted quality techniques and tools applied to the subject matter under discussion. Where such applications might not be readily apparent in the manuscript’s flow of discussion, the authors provide the necessary comments so a reasonable reader can still visualize the necessary quality techniques in use.

4. How completely does the article cover its subject matter?

0      The precise intent of the subject covered in the manuscript is unclear or ambiguous, and so the completeness and level of coverage cannot be assessed, or the sporadic nature and lack of organization of the material prevent a complete and comprehensive picture of the subject matter from emerging.

2      The manuscript presents the subject matter in an organized manner, discussing the themes and components in a manner that would leave a reasonably well-informed reader feeling that the coverage had been complete and thorough.

4      The subject coverage is comprehensive and would be interpreted by an expert in the presented discipline as complete, if not exhaustive. Portions omitted are explicitly commented by the authors or else are not material to the manuscript’s conclusions.

5. How clearly and concisely do the authors convey information?

0      The manuscript wanders and lacks clarity, making its focus and purpose unclear. The length is such that a reasonable reader would expect the same material to be offered in a much shorter length, and might reasonably abandon reading the article without actually finishing.

2      The manuscript follows a clear progression from introductory, through expository, to conclusive material. Repetition is kept to a minimum, and readers will likely be comfortable with the flow.

4      The authors convey their information in a clear and lucid order, using a minimal length that would have to sacrifice content to be made any shorter. Readability has not been sacrificed to achieve the efficiency of conciseness.

6. How logically are the conclusions presented?

0      The conclusion being drawn in the manuscript is missing or ambiguous, or no supporting premises or logic are offered that would warrant the conclusion that is offered.

2      The manuscript draws one or more conclusions with which the reader will likely agree as a result of the discussion or argument presented; however, the argument may have weak points that require supporting knowledge on the part of the reader. Readers without such knowledge might miss the point or be left questioning the conclusion.

4      The manuscript presents a sound discussion that draws warranted conclusions from a logical argument that begins with premises that a reasonable reader would consider valid.

7. How well do the authors present any limitations to their methods or conclusions?

0      The manuscript attempts to state its conclusion as a universal truth without any implicit or explicit acknowledgment of how the context of the discussion might limit or bias its use.

2      The material in the manuscript is adequate to allow readers to recognize the context within which the problems and any conclusions are drawn. Reasonable readers will not be likely to attempt to apply the conclusions or arguments to inappropriate situations.

4      The manuscript explicitly identifies any narrowness or bias contained in the discussion of the subject matter, as well as industry or topical areas where application of the manuscript’s conclusions might be unwarranted. Suggestions are offered for adapting the conclusions in those areas, or for conducting further research in the limited areas.

8. How well do the authors acknowledge and reference the works of others?

0      No references are made to other authors, publications, standards, or bodies of knowledge. A well-informed reader will likely be aware of supporting materials in the literature, or in the standards arena, that could have been referenced by the authors.

2      The authors reference other sources in general ways, particularly in support of introductory and background information, but not necessarily in support of individual distinct points throughout the manuscript.

4      The manuscript clearly and explicitly identifies the sources for all nonoriginal or industrywide materials presented, including specific citations to previous published works, and explicit general acknowledgments of bodies of knowledge and industry movements and trends.

9. How well do the authors use tables, graphs, or other visual aids?

0      The visual aids presented in the manuscript are haphazard and are not easily tied back to the theme or text of the arguments or ideas being illustrated, or else the authors have regularly failed to offer visual aids in circumstances where a reasonable reader would expect and be aided by their presence.

2      The manuscript includes tables and figures that add information and clarity to the argument or discussion being presented. Each visual aid is tied to a reasonable point in the manuscript text.

4      The authors present numerous visual aids that are appropriate and supportive of the manuscript text, and they achieve a balance between need to incorporate discussion of those aids into the manuscript text with the need to not redundantly state all of the characteristics of the visual aids in the text.