Association of American Colleges and Universities, January 2001

prepared by Ross Miller, AAC&U

Abstract

The immensely popular "U.S. News and World Report" college rankings have drawn fire from academics concerned with the practice of ranking schools according to "inputs"(student class rank, endowment, faculty credentials, etc.) rather than student outcomes. This briefing summary describes two new ratings models that focus on student outcomes and one that creates a forum for urban universities to inform the public about goals, mission, and systems important to student support. Also included is a short description of a new report that rates each state's total effort in support of higher education.

While U.S. News and World Report ranks colleges, sometimes creating the perception of large differences among schools when only very small differences in scores exist, three new models for rating colleges avoid rankings, instead concentrating on descriptions of the quality of programs and systems that have been strongly linked to student learning.

The NSSE 2000 summarized the views of 63,000 freshman and senior college students at 276 four-year colleges and universities about their institutions' contributions to student learning. The NSSE questions focus on an institution's use of resources in helping students to learn and get the most out of their college experiences, both in and out of class. NSSE was designed to offer information to students and parents about a campus's attention to factors important to student learning. NSSE may also help individual campuses improve teaching and learning by providing them with reports about their own students' engagement in learning.

Ratings are provided for five "benchmarks of effective educational practices" culled from the research literature: 1) the level of academic challenge, 2) active and collaborative learning, 3) student-faculty interactions, 4) enriching educational experiences, and 5) supportive campus environments. The benchmarks give indications of how both students and the institutions are performing. Only four institutions received ratings in the top twenty percent on all five benchmarks for both freshmen and seniors.

The Survey conveys both hopeful and disappointing findings about student engagement.

• Nearly half of all respondents reported having serious conversations with students from a racial or ethnic group other than their own.
• Nearly two-thirds of seniors reported participating in community service or volunteer work, many as part of a regular course.
• Fifty-five percent of students reported spending only one hour or less studying for every hour in class.
• Almost one fifth of first-year students never made class presentations and nearly half never discussed readings or ideas with a faculty member outside of class.

In this first year of NSSE administration, each campus was provided with its own results; results were not made available to the public. Several campuses that received especially high ratings did release their results (see www.elon.edu/e-net/nsse/academic.asp). In the future, year-to-year comparisons will be possible to indicate any changes in benchmarks that are occurring on the surveyed campuses.

As information gathered by NSSE accumulates, it will provide parents and students with important information about systems that foster learning, data now hard to come by through typical college guides. It will be in a campus's best interest to improve the systems that support student learning if they wish to improve their NSSE scores.

Commentary

NSSE does urge caution in over-interpreting these first-year results. Only 273 institutions could participate because of funding and other limits, so many fine institutions could not be designated as having exemplary profiles. NSSE emphasizes providing information for the improvement of undergraduate education and distinctly wishes to avoid institutional rankings. Plans are being made for the next survey to be conducted in Spring 2001with 330 institutions, many of which were first year participants. (For additional information, see www.indiana.edu/~nsse.)

CRI employed a survey of alumni five to six years after graduation. A series of questions gathered current information about each respondent—it did not ask them to recall what they were like in college. The questions covered the graduates' personal values, abilities, occupations, work skills, and pursuit of lifelong learning. CRI's assumptions attribute the graduates' current abilities and attitudes to their college experiences.

Data are sorted by institution and transformed into institutional profiles that show areas of apparent emphasis and non-emphasis in a college's curriculum and co-curriculum. Categories in an institutional profile include:

• Values: arts/culture, religious, civic/community, and physical fitness;
• Abilities: communicate/organize, reason quantitatively, find information; and
• Work Skills: analyzing, writing/presenting, organizing, serving customer/client.

Colleges may then be compared according to their profiles, although the information is not publicly available yet. Macalester College has chosen to post its profile on the college's web site along with an interpretation (see www.macalester.edu/~instrsch/cri/cri.htm).

The questions and format of CRI are now being used by Peterson's College Guide (www.petersons.com) in an on-line version to gather information from college graduates so that the CRI's pool of 80+ schools can be greatly expanded and the information made available to Peterson's users.

Prospective college students may take an on-line survey similar to the survey given to graduates. This generates a personal profile that can then be compared to college profiles. Presumably, a match between personal and institutional profiles indicates a good choice for college attendance. The profiling and matching appears to serve a "first sort" function for students and parents which factors individual circumstances and important learning outcomes into the college choice equation.

Commentary

The long interval between graduation and the collection of college profile data, as well as many other uncontrolled factors (graduate school, experiences before and after college, college entry profile, movement among institutions, etc.), may raise questions about linking a graduate's current attributes to attendance at a specific institution. In a description at the University of Pennsylvania's institutional research web site, the five-year period is described as a "sufficient time for the effects of their undergraduate experience to manifest, but not too far in the future to dilute those effects with intervening experiences." (http://www.irhe.upenn.edu/research/research-main.html#outcomes). It will be interesting to see if the online survey will generate the planned 500,000 valid responses to the CGI (called for in a speech by Robert Zemsky, November 28, 2000) so that accurate and useful information can be gathered on many more colleges and universities.

This Pew- and AAHE-sponsored project brings together six leading urban public universities to develop electronic institutional portfolios that will demonstrate effectiveness to several different groups of stakeholders. The selection of common student learning outcomes that will be documented on-line has been a major thrust of discussion in the past year for the six universities. While still under discussion, four outcome areas have emerged as strong candidates for portfolio inclusion: 1) communication, 2) critical thinking/problem solving, 3) a sense of civic responsibility, and 4) an appreciation for pluralism and diversity.

Defining the unique character and role of urban universities has also been a focus of the project. Participants have tentatively chosen four characteristics that, together, define urban public universities:

• student profile (older, working, part-time, first generation, diverse);
• program mix (professional, interdisciplinary, research, focus on urban issues);
• use of the city as a focus for teaching, research and service; and
• commitment to access.

Commentary

The portfolios are currently under development. A viewing of the partially completed portfolios reveals different approaches at the six schools. Such differences will make comparisons between campuses difficult for parents and students. At the moment, any agreement that the six institutions may have reached concerning the presentation of student outcomes is obscured in the individual structures of the portfolios. To facilitate the collection and comparison of information, it seems like the participants in UUPP should have a similar interface for their portfolios with campus-specific information in creative formats connected to the interface by active web links.

Just released from The National Center for Public Policy and Higher Education: Measuring up 2000

This new report from The National Center provides state-by-state "report cards" for higher education. The report focuses on the performance of each state in the key areas of 1) preparation, 2) participation, 3) affordability, 4) completion, and 5) benefits. Learning was a sixth category in which all states received an incomplete since, strangely, there were no measures available for a meaningful assessment of college learning. "All states lack information on the educational performance of college students that would permit systematic state or national comparisons," says the report (p. 23).

Measuring up 2000 is intended to provide state leaders and other stakeholders with objective information to evaluate and improve higher education. Support of both public and private colleges was considered, since private colleges play a crucial role in the overall system of higher education. The fact that the first indicator— preparation—is a measure of the strength of the high schools in a state brings attention to the need for better cooperation between the higher education and K-12 systems. The report cards do not soft-pedal deficiencies. Even North Carolina (the home state of James Hunt, the Chair of the Board of directors of the National Center and a national leader in public school reform) received a D in participation and a D+ in benefits. Texas received three Cs, one D+, and a D.

The full report is available at www.highereducation.org or may be ordered by calling 1-888-269-3652.