PROCESS

1. National Conversation

Between January 2001 and June 2002, nine meetings of the National Conversation were held at AAU universities across the U.S. More than 400 faculty and administrators from twenty institutions, representing every academic discipline, participated in these one and two day meetings.

University administration at each institution supplied the names and contact information of the faculty members on their campuses that taught or worked closely with freshmen and entry-level students. Standards for Success (S4S) staff invited these individuals and mailed them written materials about the Standards for Success project. Faculty members from the hosting university were trained to facilitate the faculty groups. S4S supplied facilitators at institutions where faculty members were not available to serve in this capacity.

At the meetings, participants were divided into separate groups according to their academic discipline. University administrators were asked to choose the discipline most closely aligned with their training or their current position. In these small group sessions, facilitators asked faculty members and administrators to describe what it takes to succeed in their university's entry-level courses. Their answers were written, audio-taped and video-taped. These sources of data were analyzed by S4S researchers and resulted in Knowledge and Skills for University Success. See the Data Collection section for additional sources of data collected. See Analysis section for description of analytic methods.

2. Data Collection

In the small group meetings of the National Conversation, faculty and administrators described what they expect from entry-level students enrolled in their courses. Facilitators gave participants actual student work samples submitted by professors at AAU universities. Participants were asked to review the samples and to describe how representative they were of what they identified as key knowledge and skills for university success. Facilitators recorded their answers on paper and the conversations were audio-and videotaped.

Participants were also given blind copies of states' K-12 standards and assessment in their respective discipline. We asked them to critique the documents and score state standards as they related to identified university expectations (not useful, useful, important, and critical), Finally, they did a cross comparison and identifed which state standards best met what they felt students needed to know and be able to do to succeed in college.

Facilitators' written notes, audio and video recording of the discussions, reviews of student work samples and written critiques of state educational standards became the core of our data sources and were analyzed to produce the Knowledge and Skills for University Success document.

3. Analysis

These sources of data gave the Standards for Success (S4S) analysts volumes of data that were broad and deep. Data analysis began with a review of all the data (approximately 5000 pages of text) to identify general statements and recurring themes as they relate to content knowledge and general cognitive skills within each individual discipline and across disciplines. Content was analyzed to ascertain the recurrence and frequency of key terms and concepts. These were tallied and their location identified. This information was logged into a database program for reference and validation via subsequent analyses. Once themes were identified, the data were re-read and sorted by identified theme. These preliminary analyses were circulated to participants in the National Conversation and they were asked to review and comment upon the accuracy of the initial interpretations of the data after each of the first five meetings.

Next the Standards for Success researchers employed a modified form of the professional judgment research technique. The professional judgment technique employs continued reviews by experts; in this case, academic faculty in their respective fields of expertise. Through multiple drafts, reviews, revisions and new drafts, these multiple editions of the findings were reviewed by additional faculties at each university until this process no longer yielded significant disagreement or suggestions for change.

What became clear after the initial five, and the first two review, meetings of the National Conversation was that new themes and vocabulary were not emerging. Distinctions were often being made among ways to state a standard, but the content was largely consistent. The final four campus-based review sessions yielded progressively fewer suggestions for significant change and more comments related to formatting. This indicated that the professional judgment process had yielded its likely maximum consensus.

4. Review

After Standards for Success (S4S) researchers collected and analyzed data collected at the first five meetings of the National Conversation, drafts of Knowledge and Skills for University Success (KSUS) were presented to participants at the last four meetings of the National Conversation. Incorporating the suggestions of these faculty participants, S4S researchers made changes to the document after each of these four meetings. As the meetings progressed, it became clear that fewer changes were being recommended. By the last two meetings, faculty participants were recommending only changes to the organization of the document with few substantive content changes.

Next, consultants at the Mid-continent Research for Education and Learning, a nationally recognized standards-setting organization, reviewed and recommended changes on drafts of KSUS.

Finally, the Content Review Panels convened in Eugene, Oregon to review KSUS. These six panels each included four to seven academic experts selected from among the National Conversation participants. In this review, the panelists debated, edited and finalized the documents, then selected individual student work samples to illustrate each KSUS standard. The resulting KSUS and corresponding student work samples are presented in the publications Understanding University Success and University Work Samples.

5. Standards

The data revealed that higher education faculty and administrators do not think about standards in the same way that state content standards writers do. Faculty statements tended to be general, rather than specific. To add greater specificity, several converging strategies were employed:

1. Course syllabi from entry-level university courses were analyzed to determine the actual knowledge and skills required to succeed in the course.

2. The staff at the Mid-continent Research for Education and Learning conducted an expert review that compared the Knowledge and Skills for University Success standards to standards from national content organizations and leading states to ascertain any discrepancies or possible omissions.

3. The draft Knowledge and Skills for University Success standards were utilized in a pilot analysis of two state tests by trained raters in order to ascertain their clarity and utility.

4. A Content Review comprised of professors who had participated in National Conversation meetings was convened to conduct the final substantive review of Knowledge and Skills for University Success.

5. The draft Knowledge and Skills for University Success standards were employed in a second analysis of assessments from fourteen states and were then edited to improve clarity and utility, based on feedback from trained raters.

The results of this process are included in the final version of Knowledge and Skills for University Success.