Results

SOFTWARE SYSTEM DOCUMENTATION PROCESS ASSESSMENT RESULTS

This section reports, for each version, the degree of key practice satisfaction from the assessments conducted for that version. For full details of assessment results click here.
 

Using Version 1


This first version was released in March 1993. It was used to assess 26 projects at 7 companies in the time period March 1993 - September 1995, and the next two tables show the results in terms of degree of satisfaction for each key practice and the overall results in terms of maturity levels.

Key Practices and Degrees of Satisfaction – DPMM Version 1

Level
 Key Practices
Not
Partial
Fully
1
 1. Consistent creation of basic software development documents
4%
77%
19%
  2. Documentation generally recognized as important
12%
8%
81%
2
 3. Written statement or policy about importance of documentation
38%
12%
50%
  4. Adequate time and resources for documentation 
27%
54%
19%
  5. Adherence to documentation standards
19%
69%
12%
  6. Use of check-off list of required documentation
31%
27%
42%
  7. Use of simple documentation tools
4%
8%
88%
3
 8. Accuracy and reliability of documentation
0%
62%
38%
  9. Mechanisms to update documentation
65%
31%
4%
  10. Mechanisms to monitor quality of documentation
65%
27%
8%
  11. Methods to assess usefulness of documentation
77%
15%
8%
  12. Use of common sets of documentation tools
35%
23%
42%
  13. Use of advanced documentation tools
65%
23%
15%
  14. Documentation-related technology and training
100%
0%
0%
4
 15. Measures of documentation process quality
100%
0%
0%
  16. Analysis of documentation usage and usefulness
96%
4%
0%
  17. Process improvement feedback loop
69%
23%
8%
18. Integrate CASE and documentation tools 73% 12% 15%

Breakdown by Maturity Level - DPMM Version 1

Level
%
1
62%
2
38%
3
0%
4
0%

Using Version 2

This second version was released in September 1995. It was used to assess 34 projects at 13 companies in 3 countries in the time period September 1995 - September 1996.The next two tables show the results in terms of degree of satisfaction for each key practice and the overall results in terms of maturity levels.

Key Practices and Degrees of Satisfaction - DPMM Version 2


Level
Key Practices
VL
L
M
H
VH
1
 1. Consistent creation of basic software development documents
0%
24%
53%
15%
9%
 
 2. Documentation generally recognized as important
29%
6%
3%
12%
50%
2
 3. Written statement or policy about importance of documentation
26%
12%
12%
9%
41%
 
 4. Adequate time for documentation 
18%
24%
26%
32%
0%
 
 5. Existence of documentation policy or standards
21%
21%
32%
18%
9%
 
 6. Adherence to documentation policy or standards
65%
26%
3%
6%
0%
 
 7. Mechanism to check that required documentation is done
15%
26%
24%
15%
21%
 
 8. Use of simple documentation tools
0%
0%
9%
32%
59%
3
 9. Accuracy and reliability of documentation
0%
18%
56%
26%
0%
 
 10. Mechanisms to update documentation
65%
26%
9%
0%
0%
 
 11. Methods to monitor quality of documentation
59%
32%
6%
3%
0%
 
 12. Methods to assess usefulness of documentation
85%
12%
0%
3%
0%
 
 13. Use of common sets of documentation tools
9%
12%
21%
24%
35%
 
 14. Use of advanced documentation tools
53%
41%
3%
3%
0%
 
 15. Documentation-related technology and training
65%
26%
6%
3%
0%
 
 16. Documentation is traceable to previous documents
53%
29%
15%
3%
0%
4
 17. Measures of documentation process quality
71%
26%
3%
0%
0%
 
 18. Analysis of documentation usage and usefulness
85%
12%
3%
0%
0%
 
 19. Process improvement feedback loop
56%
41%
0%
3%
0%

Breakdown by Maturity Level - DPMM Version 2

Level
%
1
68%
2
29%
3
3%
4
0%

 

Using Version 3

This third version was released in September 1996. It was used to assess 19 projects at 9 companies in 2 countries in the time period September 1996 – December 1998. See the next two tables.

Key Practices and Degrees of Satisfaction - DPMM Version 3

Level
 Key Practices
VL
L
M
H
VH
1
 No key practices – in this level there is only a set of basic practices
0%
6%
28%
50%
17%
2
 2.1. Existence of documentation policy or standards 
6%
11%
33%
17%
33%
  2.2. Mechanism to check that required documentation is done 

2.3. Adherence to documentation policy or standards
0%

33%
22%

39%
17%

0%
39%

17%
22%

11%
3
 3.1 Existence of a defined process for creation of documents
17%
39%
17%
28%
0%
  3.2 Methods to assure quality of documentation
6%
56%
22%
17%
0%
  3.3 Assessment of usefulness of documentation
0%
33%
50%
17%
0%
4
 4.1. Measures of documentation process quality and usefulness
39%
44%
11%
6%
0%
  4.2. Analysis of documentation process quality and usefulness
67%
28%
6%
0%
0%
  4.3. Process improvement feedback loop 
50%
22%
22%
6%
0%

 

Breakdown by Maturity Level - DPMM Version 3

Level
%
1
72%
2
28%
3
0%
4
0%

Using Version 4

This fourth version was released in December 1998. It has been used to assess 12 projects in the time period December 1998 – present. Next table shows the results in terms of degree of satisfaction for each key practice. No maturity levels are computed for this version.

 Key Practices and Degrees of Satisfaction - DPMM Version 4

Key Practices
VL
L
M
H
VH
1. Creation of basic software documents
0%
8%
25%
17%
50%
2. Management recognition of importance of documentation
0%
8%
25%
0%
67%
3. Existence of documentation policy or standards
17%
8%
25%
0%
67%
4. Monitor implementation of policy or standards
8%
8%
42%
25%
17%
5. Existence of a defined process for creation of documents
25%
0%
33%
33%
8%
6. Methods to assure quality of documentation
8%
25%
42%
25%
0%
7. Assessments of usability of documentation
8%
67%
17%
18%
0%
8. Definition of software documentation quality and usability measures
67%
17%
8%
8%
0%
9. Collection and analysis of documentation quality measures
42%
42%
17%
0%
0%
10.Collection and analysis of documentation usability measures
100%
0%
0%
0%
0%
11. Process improvement feedback loop
42%
33%
17%
0%
8%

Main Conclusions

Our results show a clear status of immaturity and general software documentation process practice non-satisfaction. Not surprisingly, this is reflected in a steady decrease in key practice satisfaction expected in more mature organizations. The key practices are listed in a natural maturity order that clearly show the trend. Assessed organizations mostly satisfy key practices linked to assuring the existence of policies or standards, but they fail for higher maturity key practices related to assuring the actual monitoring of compliance. There is an acknowledgement of the problem but little action. The results also highlight the low satisfaction for practices related to assuring the quality and usability of the actual documents produced. The main challenge is to take a serious approach to the improvement of the software documentation process, adopting and carrying out explicit policies aimed at process improvement.

One crucial point that our experience has made very clear is the importance and fundamental role of the key practices. They drive the generation of the assessment questionnaire. When an organization’s process is assessed relative to the model, the assessment is actually measuring the organization’s degree of satisfaction of the key practices. That is, how well is the organization carrying out the key practices. In spite of this, the maturity level has been the most commonly used indicator of the organization’s process maturity. The maturity level is such a broad indicator that it is of limited value as it provides incomplete information about which key practices are satisfied and which are not.


Last Updated: Thursday, April 13, 2000
by Curtis R. Cook & Marcello Visconti
© 2000 OSU Computer Science Department & UTFSM  Departamento de Informatica

Back to main page