Diversity at the UW

Part 4: Assessment

https://www.washington.edu/diversity/faculty-advancement/handbook/toolkit/Two key components help ensure fair and effective assessment of job applicants:


1. A clear and consistent assessment rubric (i.e., the criteria by which committees evaluate applicants’ qualifications), and


2. A clear and consistent assessment plan (i.e., the process by which committees evaluate applicants and make selections).

 

DOWNLOAD DOCUMENTS:

QUESTIONS? CONTACT:

CREATING AND IMPLEMENTING AN ASSESSMENT RUBRIC

An assessment rubric ensures that all applicants are subject to the same evaluation criteria, and that members of search committees apply selection criteria consistently.  Moreover, assessment criteria should reflect statements made in the job advertisement.

Ideally, the entire unit should participate in the creation of an assessment rubric to ensure that  the unit’s values are reflected in the assessment criteria.  Minimally, the search committee should be assisted by unit leadership and the unit’s diversity committee.  An assessment rubric requires the committee and the unit to define selection criteria up front, preferably while writing the job ad but always before the committee begins reviewing applications.

An assessment rubric also helps the committee and the unit clearly rank its selection criteria in terms of unit priorities—including the unit’s commitments to diversity, equity, and inclusion.

Some questions to consider:

  • What are the goals for this hire in terms of research, teaching, service, and outreach?
  • How is a commitment to diversity, equity, and inclusion a factor in each goal?
  • How does the unit rank these goals in terms of first and second priorities?
  • What types of evidence will demonstrate achievement or future potential in each area?
  • Does the job ad request materials appropriate to the assessment criteria?

Committees should consider how many distinct criteria will be useful in their assessment, and they should consider what kind of scale to employ.  Interfolio allows evaluators to use a “star” rating system; evaluators can assign between one and five “stars” for each criterion.  Some typical scales include:

  • A simple choice of “High,” “Medium,” and “Low” rankings (using only the first three “stars” in Interfolio).
  • A more elaborate choice of “Excellent,” “Good,” “Neutral,” “Deficient,” and “Unable to judge” rankings (using all five “stars” in Interfolio).

A range of sample assessment rubrics are available in the Toolkit.

OPEN RANK SEARCHES

If the unit is running an “open rank” search (i.e., “assistant or associate,” “associate or full,” or open to all three ranks), the committee should consider creating more than one assessment rubric, since different levels of achievement may be expected from applicants at different stages of their careers (e.g., in terms of research productivity, leadership, or national service).

USING THE ASSESSMENT RUBRIC AS A TOOL FOR DISCUSSION

Committees may be tempted to use the assessment rubric as they might use a rubric designed for grading coursework or reviewing grant proposals: to rank applications based on total scores. It is important to stress, however, that the assessment rubric is a tool to help maintain consistency and fairness in the review process, that is, to minimize bias either in favor of or against particular applicants. The rubric is not a substitute for active committee deliberations.

Committee members should come to meetings prepared to discuss the relative merits of specific applicants, and the review process should allow committee members opportunities to discuss any applications they find have merit, regardless of assigned scores or rankings.

CREATING AND IMPLEMENTING AN ASSESSMENT PLAN TO WORK AGAINST BIAS

Before any applications are reviewed, the committee should have agreed upon an explicit plan for how it will conduct its business in a fair and consistent manner.  Some questions to ask:

  • When will the committee begin reading and assessing applications? As applications come in?  Or after the priority deadline?
  • Should all committee members read and assess the same materials at the same stage of the search process?
  • How will committee members define and then handle potential conflicts of interest, such as a prior relationship with an applicant or with an applicant’s adviser? This issue can be especially challenging if the pool includes internal applicants.
  • By what process will the committee come to a decision about its short list? Will members vote, for example, or work to achieve consensus?
  • At what point in the process will the committee review or request references?
  • Will the committee conduct preliminary interviews? If so, will these be on site at a conference, over the phone, by Skype, or by some other electronic means?
  • By what process will the committee create its list of finalists to invite to campus?
  • How will the committee organize campus visits?
  • By what process will the committee make its final assessments and recommendations?
  • How will the committee communicate with applicants and with the larger unit at each stage of the process?

IN ADDITION TO THE CRITERIA DETAILED IN YOUR ASSESSMENT RUBRIC, SOME KEY ISSUES TO CONSIDER:

“Early Bird” Bias.  Beware of over-valuing applications that arrive early in the process, or simply giving them more attention.  It can be helpful to wait until the priority deadline before reading any applications, and to organize applications by some method other than order of arrival.

“Moving Target” Syndrome.  Beware of changing the requirements for the position as the search proceeds in order to include or exclude particular applicants.  The terms of the job ad and the criteria of the assessment rubric should be consistently applied.  It may be helpful to designate a point during the process to evaluate the usefulness of the assessment criteria and the consistency of their application.  How well are the criteria and the process working?

“Known Quantity” Bias.  Internal applicants—whether current graduate students, recent graduates, post-docs, lecturers, or part-time instructors—can be both disadvantaged and advantaged during the hiring process.  It is important to openly discuss the challenge of maintaining fairness, collegiality, and confidentiality when internal applicants are part of the pool.

Implicit Bias.  All of us are affected by unconscious bias, the stereotypes and preconceptions about social groups stored in our brains that can influence our behavior toward members of those groups, both positively and negatively, without our conscious knowledge.

One well-documented example is our tendency to feel more comfortable with those we perceive as “just like us” (so-called in-group favoritism), and numerous studies show that in situations of evaluation, members of dominant groups are typically rated more highly than others, even when credentials are identical.  This occurs regardless of the evaluator’s background—male or female, majority population or racial minority.  “Positive bias” often manifests as favoritism; “negative bias,” on the other hand, often manifests not as overt hostility but rather as a kind of neglect, as an absence of care, assistance, or attention.

It is therefore crucial to consider the potential impact that implicit bias may have on the evaluation process.

Some factors that can trigger implicit bias against particular applicants, whether or not they meet advertised selection criteria:

  • Non-traditional career paths.
  • Non-traditional research interests or methodologies.
  • Degrees from less historically prestigious institutions.
  • Prior work experience at less prestigious or lower-ranked institutions.
  • Do not appear to “fit” the unit’s existing profile (e.g., in terms of gender, age, background, interests, and so forth).

Some factors that can trigger implicit bias in favor of particular applicants, whether or not they meet advertised selection criteria:

  • Traditional career paths.
  • Traditional research interests and methodologies.
  • Degrees from historically prestigious institutions.
  • Prior work experience at prestigious or highly-ranked institutions.
  • Appear to “fit” the unit’s existing profile (e.g., in terms of gender, age, background, interests, and so forth). This is sometimes referred to as “cloning”—replicating the current unit profile in new hires.

Implicit bias is more likely to affect our decision making when we are tired, in a hurry, feeling overworked or distracted, or uncertain of exactly what we should do—in other words, under the typical conditions of serving on a search committee.  And research shows that bias can be contagious; we are more likely to feel, express, or enact bias after witnessing it in others.

Attention to implicit bias can help committees to acknowledge the value of applicants who are less obviously “like us” and thus to consider their possible positive contributions to the unit.  It can also encourage committees to openly discuss how members define concepts like “merit,” “quality,” and “excellence.”  Does the committee assume that these and related concepts have singular definitions?  And does the committee assume that definitions for these concepts are the same for all members?

Resources and case studies about implicit bias are available in the Toolkit.

IN SUM, IT IS IMPORTANT TO CONSIDER:

  • At which stage(s) of the assessment process will you apply the assessment rubric?
  • How will you ensure that agreed upon criteria are applied consistently for all applicants—including internal applicants—at all appropriate stages of the assessment process?
  • How will you work to minimize the potential impact of implicit bias?

PRELIMINARY INTERVIEWS

In many fields it is conventional practice to conduct preliminary interviews with a “long” short list—perhaps 8 to 10, or up to as many as 15 candidates—before determining which 2 to 4 to bring to campus as finalists.  To help make interviews consistent, fair, and effective:

  • Avoid offering “courtesy” interviews to internal or other applicants who do not meet stated criteria.
  • Conduct all interviews in the same format and under similar conditions—whether in person, over the phone, or over Skype—including interviews with internal candidates.
  • Have the same committee members present for all interviews.
  • Ask the same set of standard questions, in the same order.
  • Ask questions about diversity, equity, and inclusion of every candidate.
  • Make sure all interview questions comply with federal and state hiring laws and university policies. (These are available on the EOAA website.)

A guide to “fair” and “unfair” inquiries, sample interview questions that highlight issues of diversity and inclusion, and a guide to interviewing candidates with disabilities are available in the Toolkit.

ON-CAMPUS INTERVIEWS

The on-campus interview is a component of the assessment process, but it is also the beginning of the recruitment process, and thus it should involve not only the search committee but also the larger unit, the college or school, and your campus and community allies.

HOSTING THE CAMPUS VISIT

The campus visit allows finalists to showcase their professional and scholarly pursuits; it is also an opportunity for the unit to make finalists feel welcomed in a new community.

In addition to the traditional job talk, research seminar, and/or teaching demonstration; meetings with the chair or director, other department leaders, and graduate students; meals with colleagues; a meeting with the appropriate dean or chancellor; and a tour of the campus, elements of a campus visit should include:

  • Providing finalists a detailed itinerary, as far in advance as possible. To ensure equitable treatment, all itineraries should be similar, including those for internal candidates.
  • Introducing finalists to relevant faculty, staff, students, and administrators within and outside the unit with whom they might share research, teaching, service, and/or outreach interests. How can you help finalists imagine local professional networks?
  • Asking finalists if they would like to visit relevant research centers, facilities, or other campus resources, and/or to meet with a human resources or benefits officer. It is best to create a list of resources finalists can review before they travel to campus.  A sample list of campus resources is available in the Toolkit.
  • Providing venues for finalists to ask questions they might not feel comfortable asking members of the unit (e.g., about partner hiring, family or medical leave, stopping the tenure clock, disability accommodations, resources for childcare or eldercare, unit or campus climate toward women and minorities). The meeting with a dean can be an opportunity for these kinds of questions if it is clear they can be asked in confidence.
  • Maintaining clear and open communication with finalists. It is important to be honest about expectations, as well as about issues of funding, space, or other resources.
  • Explaining the unit’s and the university’s expectations about teaching, research, service, and the promotion and tenure process.
  • Introducing finalists to relevant college and campus resources for their success.

CAMPUS VISITS AND INTERNAL CANDIDATES

If the list of finalists includes internal candidates, it is important to:

  • Insure that the itineraries for their campus visits are as similar as possible to those of external candidates.
  • Be intentional about maintaining fairness, collegiality, and confidentiality.
  • Inform internal candidates about the campus visit process.
  • Encourage internal candidates not to attend public events, such as job talks or open meetings, involving the other finalists.

A best practice is to host internal finalists first in order to avoid any potential perception that internal finalists have an advantage from having seen firsthand or heard about the other finalists’ visits.

NEXT SECTION: RECRUITMENT