Ernst v. City of Chi. - 837 F.3d 788 (7th Cir. 2016)

Rule:

The technical standards for validity studies are set forth in 29 C.F.R. § 1607.14(B)(4). This section requires that the volunteers in a study's sample population should, as far as possible, be representative of the candidates normally available in the relevant labor market for the job. 29 C.F.R. § 1607.14(B)(4). Representativeness is a fact-sensitive inquiry.

Facts:

The Chicago Fire Department employs several hundred paramedics. When hiring new paramedics, Chicago has not always tested its applicants' physical skills. From the 1970s through the year 2000, paramedics were hired without any physical test. However, in 2000, Chicago implemented a physicals-skills test created for it by Human Performance Systems, Inc., led by its president Debora Gebhardt. Gebhardt had previously created a physical test for the Chicago Fire Department’s entry-level firefights which had a disparate impact on women. Regardless, she was rehired for the paramedic test without taking bids from anyone else. After they were denied employment based on their exam results, Ernst and her fellow plaintiffs filed suit. They challenged the skills test as discriminatory; they urged that there was no evidence of Chicago paramedics ever lacking the physical ability to properly care for their patients. Instead, they argued, the test was implanted to keep women out. Ultimately, their suit had two parts: disparate-treatment claims, and disparate-impact claims. The district judged ruled for the defense, despite the jury asking for clarifications on the instructions given to it.

Issue:

Did the district court err in finding that Gebhardt’s job analysis and physical-skills study satisfied the express federal regulations on validity studies?

Answer:

Yes.

Conclusion:

Re: whether study sample are representative individuals — By Gebhardt's own testimony, however, the subject volunteer paramedics—who were asked to volunteer, rather than randomly chosen—did not represent the skill-set in the general population of Chicago paramedics. Gebhardt testified that the Chicago volunteers performed better than public-sector and private-sector paramedics normally perform. In her formal report on this study, Gebhardt forthrightly stated that "the Chicago EMS personnel were found to be above average. This was especially true for the women." Because she was concerned that the results of her Chicago study might not be representative, she combined her data on 52 Chicago paramedics with a comparable data-set on 87 New York City paramedics. As Gebhardt's study report stated, and Chicago affirms in its brief, she did this "[t]o avoid artificially inflating the passing score." In other words, the New York paramedics presumably had lower scores than the Chicago paramedics, which helped draw the average score toward a more normal performance level. For the combined Chicago and New York City scores to result in a truly normal or average score, however, the New York City paramedics' scores would have to be significantly lower than normal. With 52 Chicago paramedics and 87 New York City paramedics, Gebhardt had a total sample population of 139 paramedics. And with this small sample size, the 52 Chicago paramedics constituted 37% of the total sample population. About four in ten paramedics, in the overall sample population, were still from Chicago. This is still a sufficiently high percentage for Chicago paramedics to pull up the average scores and skew the overall results. Chicago does not explain why it believes that, by adding 87 New York scores, the problem of abnormally high scores was resolved. Perhaps this is a problem that could be resolved by comparing the combined Chicago and New York results with the studies that Gebhardt considered, when she concluded that the Chicago volunteer paramedics had abnormally high scores, and showing that the combined results are comparable. But the record does not indicate that even the combined sample population represents the general paramedic population.

Re: whether test focused on primary skills learned on the job — The entrance exam tests three skills: the modified stair-climb, arm endurance, and leg lift. These skills were validated by correlating each skill with all three work samples: the lift and carry, stair-chair push, and stretcher lift. There is a statistically significant correlation between these physical skills and these work samples. In this, the study is fine. On the issue of reliability, however, the lift and carry poses a problem: its reliability score is only 0.503. That is a 50/50 chance of reliability. Federal regulations direct that reliability, in validity studies like this one, "should be a matter of concern to the user." This is particularly true when the reliability of this lift and carry is equivalent to the proverbial coin toss: heads, it is reliable; tails, it isn't. Further, there was no apparent effort to separate the lift and carry from the rest of the study. Because each of the three skills (modified stair-climb, arm endurance, and leg lift) was validated by correlating it against all three work samples (lift and carry, stair-chair push, and stretcher lift), the unreliability of this lift and carry undermines all three skills that Chicago tests in its physical entrance exam. "All tests must be statistically examined for evidence of reliability before the test developer can establish the validity of the test.” Given the lack of reliability in her study, the test developer in this case cannot establish the validity of her study. Even if reliability was fully established in this case, validity would be a problem. The plaintiffs legitimately question whether the work samples themselves are a valid measure of job skills. The problem here is that Chicago used the work-sample tests to validate the skills tests—without ever validating the work samples. As a result, we cannot conclude that these work samples reflect "the primary focus of" paramedic skills learned on the job. In comparing the skills that Chicago paramedics learn on the job with the skills that Gebhardt's three work samples require, these are two different sets of skills. Even if they were the same, the work-sample skills are more taxing than real on-the-job skills. Gebhardt has test subjects cycling abnormally heavy stretchers up and down for 4 minutes and 20 seconds, for example, even with rest times omitted. In contrast, paramedics usually transport relatively lighter stretchers across a distance of 100 feet or less, which should require substantially less than 4 minutes. And as one of the court’s sister circuits has affirmed, plaintiffs should prevail on their disparate-treatment claim when a physical-skills entrance exam is "significantly more difficult than the actual job workers performed at the plant." The difficulty of these work samples also undermines Chicago's argument that these work samples represent real skills that Chicago paramedics learn on the job.