Few would disagree that the "quality" of undergraduate and graduate programs, though heretofore loosely defined, has been a concern within academia. Until recently, however, account-ability, in the form of documenting the achievement of academic outcomes and goals, has been relatively low. The formal assessment of quality in academia is now becoming a major undertaking at most universities ( El-Khawass, 1987 ; Ewell, 1985 ). Documentation of outcomes in the areas of teaching, research, and service have been mandated by state and regional accreditation agencies (e.g., Southern Association of Colleges and Schools [SACS], 1987 ). The development of outcome assessment plans and their implementation, while providing increased accountability to the public, will also provide a means for evaluating whether goals are being attained in specific academic programs and for evaluating the relative quality of instruction and scholarship at various institutions.

Within the field of psychology, the American Psychological Association (APA) accreditation process provides information on the quality of doctoral programs in clinical, counseling, and school psychology. However, there exist little data regarding the quality of master's-level psychology programs. These programs provide graduate training to thousands of students each year by preparing students for jobs requiring the degree within psychology or related fields, or by preparing students for graduate training at the doctoral level, or both. The primary purpose of this study was to provide information regarding one aspect of "quality" among faculty belonging to terminal master's degree programs in psychology: research productivity.

Two approaches have been used to examine the quality of academic programs: reputational ratings and research productivity assessments. In evaluations of program quality based totally, or in part, on reputational ratings ( Cartter, 1966 ; Jones, Lindzey, & Coggeshall, 1982 ; Roose & Andersen, 1970 ), researchers have typically asked for impressions regarding the scholarly quality of a program's faculty. Problems with relying on this type of subjective evaluation as the sole criterion for determining quality have been delineated elsewhere ( Cox & Catt, 1977 ; Thoresen, Krauskopf, & Cox, 1975 ).

As an alternative, assessments of program quality based on research productivity have become a more objective counterpart to the traditional subjective evaluation procedures ( Cox & Catt, 1977 ; Howard, Cole, & Maxwell, 1987 ). Such assessments have been conducted separately for counseling psychology ( Howard, 1983 ), industrial/organizational psychology ( DeMeuse, 1987 ; Howard, Maxwell, Berra, & Sternitzke, 1985 ), quantitative psychology ( Maxwell & Howard, 1986 ), and social psychology ( Gordon & Smith, 1989 ) and for psychologists based in Veterans Administration (VA) medical centers ( Cox & Thornton, 1988 ). However, given that the majority of research published in the journals examined in these studies has been produced at doctorate-granting institutions, previous examinations have been primarily focused on doctorate-granting degree programs.

The present assessment was restricted to schools offering master's-level programs in psychology, and thus the research productivity of the psychology faculty at comprehensive level institutions could also be measured. Such information would allow faculty at these institutions to view their program's relative level of research productivity and would also provide prospective students with data regarding the emphasis placed on research among various terminal master's programs in psychology.

Section III of the 1986 edition of Graduate Study in Psychology and Associated Fields ( APA, 1986 ) was consulted in order to create the original list of institutions to be included in the study. To be included, an institution had to have an ongoing, terminal master's-level program in applied, clinical, counseling, experimental, general, industrial/organizational, or school psychology. Those schools that offered the terminal master's degree but that also offered doctoral degrees in any area of psychology were not included in the list. This procedure resulted in a list of 165 institutions.

In order to further reduce this list, the corporate source index of the 1980—1987 volumes of the Social Sciences Citation Index ( SSCI ) was examined, and only psychology departments that showed at least 15 publications over the eight-year period (1980—1987) were included for further analyses. ¹ This procedure narrowed the original list of 165 institutions to 77 institutions.

The corporate source index in the SSCI lists publications by departmental affiliation. However, when an affiliation is not noted (e.g., no mention of departmental affiliation in the author notes of an article), the publication is listed in a general section of the corporate source listing for that institution.

General listing entries were counted as psychology department publications if the author also appeared in the psychology department listings. In addition, a list of all psychology journals indexed with the SSCI , along with a list of education and management journals in which psychologists frequently publish, was compiled. When a publication in any of these journals appeared in the general listing category but the author's name was not shown among the psychology department entries, the institution's catalog was consulted in order to determine whether the author was a member of the psychology department. If the search through the catalog revealed a departmental affiliation other than psychology (business, education, management, marketing, psychological services or clinic, etc.), this was noted and the article was not counted toward productivity credit for that institution. However, if the author's name could not be found in the catalog, the article itself was then examined in order to determine departmental affiliation or the status of the author. In many cases, such searches revealed that the first author was a student in the program, and these articles were subsequently included as productivity credit. If no information on departmental affiliation could be found, the article was not counted toward that institution's productivity credit total.

A department received 1 point for every "first-authored" journal article (one whose first author was a psychology faculty member or student). ² Entries designated as research notes or brief reports received .50-point credit. In order to calculate an institution's productivity in the APA journals, the number of first-authored publications in the 15 journals of the American Psychological Association (excluding reviews in Contemporary Psychology or letters in other APA journals) was counted for each of the 77 institutions through a search of the PsycInfo data base.

Both overall and APA productivity scores were weighted by the number of full-time faculty in a given department. Information regarding the number of full-time faculty members in the psychology departments at these institutions was gathered from the 1986 edition of Graduate Study in Psychology and Associated Fields ( APA, 1986 ). For a few cases in which this information was not provided, the 1986—1987 catalog for the specific school was consulted in order to determine the number of full-time faculty members in the department. ³

One rater coded the entries for 39 of the institutions and a second rater coded the entries for the remaining 38 schools. Twenty percent of the entries were chosen at random in order to examine the reliability of the data. The unit of analysis was 1 year (i.e., all entries within a single year at a given institution had to match for agreement). The results of the reliability analysis revealed an average interrater agreement of .88. Data on 50 institutions gathered from the PsycInfo database were also checked by a second rater. The unit of analysis was, again, 1 year, and the interrater agreement was .98.

Examination of the productivity distributions among the 77 institutions for the total number of first-authored publications and the total number of publications in the 15 APA journals revealed that both distributions were positively skewed (skewnesses = 2.54 and 1.26, respectively). A log transformation previously used by Maxwell and Howard (1986) was therefore applied to the data in order to compute a productivity index. The following formula was used:


where X _i represents the institution's APA productivity score and overall productivity score, and z and ln represent the standardizing and logarithmic transformations, respectively. A productivity index based on the average standardized scores without the logarithmic transformation P ₂ was also computed. However, the rank-order correlation between the two indexes was extremely strong ( r = .99 and t = .95), and therefore only data based on the index P ₁ are reported. An institution's composite index productivity score was also weighted by the number of full-time departmental faculty. The P ₁ index tends to reward institutions that scored high on both productivity measures.

The 20 institutions ranked highest in productivity in the APA journals are listed in Table 1 . ⁴

Rank-order correlation coefficients were computed in order to examine the relation between overall productivity rank and APA productivity rank. This assessment revealed a small-to-moderate relationship between the two productivity measures ( r = .55 and t = .40). The rank-order correlation between total productivity and the P ₁ composite index was strong ( r = .92 and t = .76).

Further examination of the data revealed that the top 50 institutions compiled 2,133.5 points of productivity credit over the eight-year period (1980—1987). Of this credit, 292.0 points (13.7%) were in APA journals. Among the top 50 schools, the number of first-authored publications per individual faculty member in APA journals during 1980—1987 was slightly less than half (0.47). However, the data also reveal a wide range of variability on this measure. The top six schools listed in Table 1 each had at least one first-authored publication per faculty member in the APA journals in 1980—1987.

A review of the institutions in Table 1 may prove to be surprising to some readers but not to others. Although no formal assessment was made, it appears that the academic reputation of the psychology departments at some of the top schools on the list matches their ranking based on the weighted APA productivity score (e.g., Bucknell University, Swarthmore College, Wake Forest University, and Wesleyan University). Comprehensive institutions from a number of state systems are also represented among the top 50 schools, including five schools in the California State system.

Although the mean weighted APA productivity score (.47) may at first glance appear relatively small, a comparison with data gathered from the top 40 doctorate granting institutions places the present data in a more appropriate context. After it was adjusted for faculty size, the mean APA productivity total at the 40 most productive doctorate granting institutions was 2.12 ( Howard et al., 1987 ). Other methodological differences between the two assessments are undoubtedly responsible for a portion of the difference between the two mean APA productivity scores.

First, the comparison is being made between a list of 40 and a list of 50 schools. The addition of the next 10 ranks in Howard et al.'s (1987) study would reduce the mean APA productivity credit score from that study. Other differences include the fact that the productivity assessment by Howard et al. (1987) was conducted over 10 years (1976—1985) as opposed to 8 years, that all APA publications regardless of departmental affiliation were counted, and that the productivity credit score was based on a formula that gave proportional credit to coauthors in multiple-authored articles. Given these substantial methodological differences, it is interesting that the top two schools in the present study (Swarthmore College and North Dakota State University) were ranked 16th and 28th, respectively, among the top 40 schools listed by Howard et al. (1987) .

Relying on the productivity indexes described in this study as the sole measure of program quality may be as inappropriate as relying on more subjective reputational assessments. The information gathered in this study does not directly address the issue of graduate training per se or the publication of chapters, review articles, or books. Maxwell and Howard (1986) emphasized the difference between productivity and the impact of what is produced. Although impact factors are available for research in a specific journal, this measure of impact averages over all the articles in a given year and thus could not be used to accurately estimate the impact of individual publications.

In addition to a more specific focus on research productivity (e.g., the number of graduate student presentations at conventions and subsequent publication of theses and dissertations), graduate students would be well advised to pay attention to additional factors related to quality of graduate training, such as the number of graduates passing licensing examinations and the number of graduates who are subsequently accepted into doctoral programs.

The shortcomings of this form of archival investigation aside, the findings reveal that there does exist a moderate relation between overall research productivity and publication in the APA journals among the most productive psychology departments at comprehensive-level institutions (which offer the master's degree, but not the doctorate, in psychology). The listing in Table 1 also suggests that there may exist a relation between reputation and research productivity among the departments included in the analysis.

Although the listing can be taken only as a single index of research productivity, it does provide prospective graduate students with information related to the degree of emphasis placed on research among master's-level psychology programs. Such information may also be related to the percentage of graduates who subsequently enter doctoral programs in psychology. Last, the results of the study provide a basis for making objective productivity comparisons among terminal master's-level programs in psychology.

1

The list also included institutions that did not meet the 15 publication criterion in the SSCI but were subsequently found to have a significant number of publications in the APA journals in relation to the number of full-time departmental faculty.

Given that research productivity from the years 1980—1987 was examined, the 1984 edition of the Graduate Study in Psychology and Associated Fields ( American Psychological Association, 1984 ) was also reviewed in order to see whether changes in faculty size had taken place. No significant changes (i.e., greater than 10%) were noted among the 77 institutions.

Table 1.