June 16, 2010

Accountability Methodology

By Undergraduate Academic Affairs

An accountability report organizes aggregate statistics for undergraduate involvement in a certain activity type. An Activity models a person’s
participation in some sort of trackable activity. The primary purpose of
the Activity models is to compile and generate
accountability statistics for the provost and the legislature. We currently
focus on two metrics for undergraduates: public service and research.

How activities are stored

Activity is subclassed by a number of Activity varieties:

ActivityCourse: for tracking an entire UW course for a particular quarter
ActivityProject: for tracking an ongoing project, such as Students in Service, that lasts
several weeks or even several quarters
ActivityEvent: for tracking a one-time event, such as the MLK Day of Service, that has a
more limited scope.

Sources

The following data sources are used to calculate accountability statistics:

  • Course lists provided by departments, including
    course_branch, course_no, dept_abbrev, and section_id for each course along
    with a Y/N for each quarter that the course should be counted. (Note: if
    section_id was not specified, we automatically pull all sections
    for that course. In some cases this will result in duplicates but they will
    be removed during duplication processing).
  • Individual student lists provided by departments
  • Carlson Center service-learning participation (as tracked by EXPO) —
    only for the public service report
  • Applicants and group members of applications whose Offering has an ActivityType defined (currently, this includes
    Mary Gates Research scholarship [research], Mary Gates leadership
    scholarship [public service], and the Undergraduate Research Symposium
    [research]).

    • For session-based events (e.g., UGR Symposium), session presenters (all
      group members) are counted.
    • For non-research-based award offerings (e.g., Mary Gates Leadership), only
      awarded applicants are counted.
    • For research-based award offerings (e.g., Mary Gates Research), all
      complete applications (mentor approved) are counted.

Intended Results

This system generates the following general types of statistics wherever
possible:

Student-quarters
The number of quarters a student spent working in the given activity. One
student for one quarter = 1 student-quarter. One student for three quarters
= 3 student-quarters. Three students for one quarter = 3 student-quarters.
Number of students
The total number of unique students who participated

Number of hours
A sum of all total number of hours that students participated

Coverage
The number/percentage of departments reporting information

Accountability Methodology

This section describes the methodology used to calculate accountability
statistics. All statistics are generated in groups based on the ActivityType of the given activity.

  1. Raw Data AccumulationFor each data source, pull relevant records. Calculate number of students
    and hours for each record.

    1. Student participation is gathered for department
      course lists:

      1. Each ActivityCourse that meets the
        requested criteria (e.g., “year = 2009”) is validated against
        the SDB to verify that it is a valid course.
      2. For each course, the class enrollment is pulled, resulting in individual Student records.
      3. Students who withdrew, failed, or did not receive credit for the course are
        removed.
    2. Student participation is gathered from
      department individual participant lists.
    3. Student participation is gathered from
      EXPO’s service-learning courses (ONLY for public service report).
    4. Student participation is gathered from
      EXPO’s application system (ONLY matching ActivityTypes)

    Final result: hash with Students as keys and an activity array for each
    quarter, e.g.:

        Student_1_system_key   =>  {
             AutumnQuarter => [ActivityCourse, MaryGatesLeadershipApplication],
             SpringQuarter => [ServiceLearningCourse],
                            },
           Student_2_system_key   =>  {
             WinterQuarter => [ActivityEvent]
           }
           Student_3_system_key   => {
             :no_quarter => [ActivityProject]   # in rare cases, we don't know what quarters a project spanned
           }
           ...
  2. Removal of non-undergraduatesFor each student in the data hash, check each quarter of activity to ensure
    that the student was enrolled as an undergraduate during that quarter.
    Non-undergraduate quarters are removed. To do this, we look at the
    student’s transcript record for that quarter.

    • If the quarter being evaluated is summer, the record is never removed, even
      if the student was not enrolled or not listed as an undergraduate for that
      quarter.
    • If there is no transcript record for that quarter and it is not summer, we
      remove it from the data hash.
    • If the “class” code for that quarter is higher than 5 and it is
      not summer, we remove it from the data hash.

    Notes:

    • For this report, “undergraduate” is any student with a class
      standing of 1, 2, 3, 4 or 5 (note that we do not include non-matriculated
      students but we do include “5th years”).
    • If removing a quarter from a student in the data hash results in that
      student having no more valid quarters, the student is completely removed
      from the data hash and will not count toward any totals.
    • Any :no_quarter listings are still analysed based on the first quarter of
      the academic year being evaluated.

    Final result: same as in step 1, but only with the students’ quarters
    that are valid.

  3. Duplication Processing
    For each student in the data hash, process each duplication rule (see rule
    definitions below). As the processing happens, records that are removed are
    saved in a duplicates array for later review.

    1. Same quarter, same course
    2. Same quarter, multiple courses
    3. Same quarter, multiple activities

    Final result: same as in step 2, but only with the activities that should
    be counted in the final tally.

  4. Statistics Calculation
    1. Student-quarters: sum of the count of each student’s Quarter objects, plus any :no_quarter listings.
    2. Number of students: size of student data array.
    3. Number of hours: sum of number of hours per student; calculated differently
      based on the data source.

      1. Course list: number of credits the student
        earned for the course, multiplied by the constant
        COURSE_CREDIT_TO_HOURS_MULTIPLIER and the WEEKS_PER_QUARTER (10).
      2. Individual list: Hours are specified by department (as either hours per
        week or total hours per quarter)
      3. Carlson Center service-learning: 20 hours per quarter is used, as
        recommended by the Carlson Center (this is the minimum requirement by the
        CC)
      4. Application processes: use hours_per_week as specified by the student, if
        available. If hours_per_week is not available, we calculate the average
        number of hours for all students in the given activity type.

        • For award-based processes, multiply the calculated-hours-per-quarter times
          the number of requested award quarters
        • For event-based processes, the number of hours is only counted for the
          quarter during which the event occured.
  5. Additional Reports
    • Department Sponsorship Breakdown: in what
      departments are these activities happening?
    • Department Cross-Pollination: e.g., how many
      BIO majors are doing research sponsored by medicine?
    • Demographic Analysis: majors, gender, class standing, and race/ethnicity of
      students participating.
    • Coverage: how many departments are represented/reported, broken down by
      college, discipline, and academic/admin.

Rules for duplicates

These rules define how we handle students who have multple records in a
given quarter. Note that activities are always
evaluated through the lens of a specific ActivityType, so these rules only apply to
duplicates within the same ActivityType.
For example, if a student participates in two research projects and one
public service course in the same quarter, the public service course is not
factored into the duplication processing of the two research projects.

Any time duplicates are removed, a Duplicates Report or Discrepancy Report
is generated to allow for manual inspection and override.

  • Same quarter, same course: If a student is enrolled in the same course for
    the same course_no (e.g., they are enrolled in both a lecture section and a
    quiz section for the same course), only the section with the larger number
    of credits are counted.
  • Same quarter, multiple courses: Students receiving credit for multiple
    courses in the same quarter are not de-duped, unless the faculty
    mentor/instructor is listed as the same for both course.
  • Same quarter, multiple activities: Students who receive credit for a course
    and are also listed under an ActivityProject or ActivityEvent for the same quarter will only
    count the larger of the number of hours.

Comments are closed.