Completed Research

Climate and Inclusion - Reliable and Valid Measures
© 2019 Mangala Subramaniam
 
For more information on this study please contact butlercenter@purdue.edu.
A note on conceptualization and operationalization of measures for climate is available here.
 
Study
We developed a pilot survey instrument to test the validity and reliability of the above discussed seven sub-dimensions of climate which comprised a total of 44 measures. We created the questionnaire in Qualtrics, a web-based survey tool. Questions were grouped by whether they referred to experience or perceptions, with experience questions being rated on a three-point frequency scale (frequently to never) and perception questions being rated on a five-point agreement scale (strongly agree to strongly disagree). We included additional demographic variables in the survey- rank, years in rank, gender, and whether subjects had attended a college-level meeting in the past year. Given our interest in testing for validity and reliability, we did not allow participants to skip questions, an option enabled by the Qualtrics software. Approval was obtained for the use of human subjects from the university’s Institutional Review Board (IRB). We used a consent form as approved by the IRB.
 
Analytic Strategy
For our reliability and validity analysis, we relied on two key statistics: coefficient alpha and factor analysis (Spector, 1992). Coefficient alpha (Cronbach, 1951) is a measure of the internal consistency of a scale. It is usually positive and takes on values from 0 to just under 1.0 where larger values indicate higher levels of internal consistency. We use Nunnally’s (1978) widely accepted rule of thumb that alpha should be at least 0.70 for a scale to demonstrate internal consistency.

Results
The final set of reliable and valid measures is listed below. Alpha (α) is reported from reliability tests. Validity of scales are based on factor analysis; factor loadings are in parentheses.

(A) Mistreatment Experience: Verbal (α = 0.86)
  • Being yelled at (0.630)
  • Being prevented from expressing your opinion in meetings (0.668)
  • Having someone try to turn others in your department against you (0.683)
  • Being the target of a derogatory verbal remark (0.731)
  • Being pressured to change opinions (0.665)
  • Being called demeaning names (0.523)
  • Humiliation in front of others (0.779)
  • Belittled (0.778)
  • Being discouraged from speaking in department meetings (0.755)
Response options were on a frequency of occurrence scale of 1 (Never) to 3 (Regularly)

(B) Mistreatment Perception: Verbal (α = 0.90)
  • Some faculty have a condescending attitude toward LGBTQ people (0.840)
  • Some faculty have a condescending attitude toward women (0.807)
  • Some faculty have a condescending attitude toward racial/ethnic minorities (0.895)
Response options were on an agreement scale of 1 (Strongly Disagree) to 5 (Strongly Agree)

(C) Bias: Implicit (α =0.84)
  • Yearly evaluations at the department level are fair (0.708)
  • Criteria for research support at the department level is uniformly applied (0.746)
  • There is adequate feedback regarding progress toward promotion (0.587)
  • My research is valued in my department (0.792)
  • My teaching is valued in my department (0.796)
  • I do not know the criteria for allocation of research support in my department (0.667)
  • My service contributions to my department are ignored (0.718)
Response options were on an agreement scale of 1 (Strongly Disagree) to 5 (Strongly Agree)

(D) Bias: Explicit (α= 0.83)
  • Research on race and gender is devalued in my department (0.671)
  • Members of my department have hostile views toward racial/ethnic minorities (0.817)
  • Members of my department have hostile views toward women (0.741)
  • Members of my department have hostile views toward LGBTQ people (0.759)
  • Women are consistently denied tenure in my department (0.640)
  • Racial/ethnic minorities are consistently denied tenure in my department (0.762)
  • My department doesn’t offer accommodation for faculty with disabilities (0.440)
  • Faculty who have needs for accommodation in my department are reluctant to bring it up (0.548)
Response options were on an agreement scale of 1 (Strongly Disagree) to 5 (Strongly Agree)

References Cited
Cronbach, L.J. (1951). Coefficient Alpha and the Internal Structure of Test. Psychometrika 16: 297-334.
Spector, P. (1992). Summated Rating Scale Construction: An Introduction. New York: Sage.
 
Research Assistant: Zak Palmer