The Pennsylvania Proposal and the Meaning of Merit

A group of faculty at the University of Pennsylvania have published a vision for the future of the institution, in the form of a proposed constitution. The document has a large number of signatories, some of whom have chosen to remain anonymous, but whose affiliations have been verified before posting.

Although there is much in the document that I would gladly and publicly endorse, I have chosen not to sign it. This post is an attempt to explain why.

The proposed constitution has three components, dealing respectively with mission and scope, hiring and promotion, and admissions and curricula. The first of these concerns free expression and institutional neutrality, and is heavily influenced by the Kalven Report. Even some of the wording is virtually identical.¹ For example:

The instrument of dissent and criticism is the individual faculty member or the individual student. The university is the home and sponsor of critics; it is not itself the critic (Kalven Report).

The agents of dissent and critical discourse within the university are the individual community members of the University of Pennsylvania. The university serves as the hosting entity for these critics, but it does not act as the critic itself (Proposed Constitution).

For reasons discussed in an earlier post, I think these principles are sound, sensible, and worth adopting. Were the proposal limited to this, I’d have gladly signed my name to it.

The remainder of the document deals with hiring, promotion, admissions and curricula. Here, again, there is plenty that I would happily endorse, such as the use of “objective and measurable criteria” for selection, as well as a focus on “rigorous training and critical analysis of multiple perspectives” in the design of curricula.

Where I part ways with the proposal is in its conceptualization of merit, which is simplistic and sidesteps thorny questions of how precisely this crucial concept should be defined, measured, and implemented in practice.

I have been thinking and writing about these issues for a while now (see, for example, these two posts and this article). When making a case for meritocratic selection, it is important to be clear about whether merit is a property of individuals or of larger groupings such as teams, and whether characteristics that are not causally related to performance (such as religion, ethnicity, or gender) may nevertheless be informative about the potential of individuals to succeed. Furthermore, merit has to be defined with respect to the mission of an organization.

According to the proposal, the core mission of the University is “the pursuit, enhancement, and dissemination of knowledge.” This involves the “cultivation of students’ critical thinking skills and disciplined curiosity.” As a consequence, all admissions decisions are to be driven by the “ability to excel” along these dimensions. The word excellence is mentioned no less than eight times.

Excellence in alignment with organizational goals is a worthy objective, but how does one identify those among a pool of applicants who are most likely to exhibit it?

Consider some possibilities.

The institution could devise (or outsource) an examination of some kind, and simply select those who score highest. This is the process used for admission to some specialized high schools in New York City, elite colleges in South Korea, and the officer ranks of the Indian Civil Service. The idea dates back to Plato and Confucian China, and this is what many people have in mind when they think of meritocratic selection. The procedure is certainly objective and transparent, and those who achieve the highest ranks have exhibited a very high degree of performance on some relevant dimensions.

But this approach has at least two drawbacks. First, it neglects information that could be valuable in interpreting scores, such as the human and material resources that were available to the various candidates over the course of their lives. And second, it gives rise to the possibility that those selected will approach problems in similar ways, duplicating what others bring to the table, and compromise the performance of teams.

These are not trivial concerns. Anita Woolley and her collaborators have been exploring the collective intelligence of groups, which is only weakly correlated with the intelligence of the group’s constituent members. As Scott Page has argued, “diverse perspectives, heuristics, interpretations, and mental models improve our collective ability to solve problems and make accurate predictions.” The diversity here refers to differences in “cognitive toolboxes” but the gender composition of teams can also affect performance.

Furthermore, even at the level of predicting individual performance, group membership can be informative about potential without being causally linked to it. This can happen because unobserved characteristics related to performance may be distributed differently across groups.

This has been known for several decades. At a 1987 congressional hearing, the Director of Admissions at MIT testified that an academic index constructed by combining high school grades, class rank, and standardized test scores was quite strongly correlated with grades in college, but that “the index underpredicts the final grade point average for women at all levels.”

Along similar lines, a study examining selection into a gifted and talented program for young children found that performance could be improved by taking account of demographic information. The authors draw the following conclusions from their findings (emphasis added):

A student who has, in some way, experienced hardship may underperform on achievement tests relative to his or her capability. By taking account of such empirically grounded differences across demographic groups, a district may be better able to determine which students are most suited to admission to the gifted program… While this profiling based on differences in distributions across racial groups is beneficial to minority students, it is not preferential treatment.

The Penn proposal requires that “no factor such as gender, ethnicity, nationality, political views, sexual orientation, or religious associations shall be considered in any decision related to student admission and aid.” The point I am making here is that this insistence is not consistent with the pursuit of excellence above all else.²

As an alternative to test-based admission, consider holistic review. This is the practice at most highly selective colleges and universities in the United States today. As currently implemented, holistic review takes into account a very broad range of information about candidates, including markers of identity such as gender and ethnicity.³ It is possible to conduct holistic review while remaining blind to such factors, and this seems to be a direction in which many universities are moving. But the approach is neither objective nor transparent, and would therefore probably fail to satisfy the authors of the Penn proposal.

What about percentage plans, which guarantee admission to those who rank highest in their particular high schools, rather than in the pool of applicants taken as a whole? Such plans are in place for admission to public universities in California, Florida, and (most famously) Texas. This approach takes account of the fact that a given test score may have a different meaning depending on the human and material resources that were previously invested in a student. It is also objective and transparent (though possibly subject to gaming).

Should private universities modify their selection criteria to incorporate some variant of this idea? One possible objection is that percentage plans do not, in fact, result in selection for excellence. Someone just outside the top ten percent in a highly competitive school may have significantly greater potential for success than someone within the top ten percent at a poorly resourced school.

There are two possible responses to this concern. One is empirical—it turns out that at least in Texas, those who gained eligibility for admission under the plan were more likely to enroll and graduate from college, while those who were displaced by it did not experience reduced graduation rates or earnings. That is, the plan appears to have helped some students without imposing comparable costs on others.

The other response to the objection is more fundamental. The idea that our best universities should admit only those who have already exhibited the greatest potential for success is a bit like our best hospitals admitting only the healthiest patients—Adonis with a pimple rather than the desperate heart patient in the language of this satirical article. Perhaps a greater emphasis on value added rather than absolute accomplishment is warranted.

I don’t think it would be a terrible thing for Penn and its peer institutions to adopt something like a percentage plan, with lotteries to deal with the large number of eligible applicants. There is a great deal of randomness already in the selection process, with very little to distinguish students who are accepted from those who are (barely) rejected. It wouldn’t hurt to make the role of chance explicit.

Merit must be defined in relation to organizational mission, and missions change over time. Take a close look at this picture of Columbia’s iconic Low Library, and you’ll see a mission statement inscribed in the text below the dome:

Do you see it? For the Advancement of the Public Good and the Glory of Almighty God. Those words were inscribed in 1897, and the mission has certainly evolved since then. It is not (despite appearances) set in stone. As we rethink our selection practices, perhaps it is an opportune time to also rethink our mission in this world.

1. It would have been appropriate for this dependence to have been explicitly acknowledged, perhaps with a link or note, otherwise this comes perilously close to plagiarism. I don’t think the oversight was intentional, and it reveals just how easily one can slip up in this way.

2. Taking account of demographic information for performance reasons need not always benefit groups viewed as historically disadvantaged. The removal of a quota favoring male applicants to a teacher training program in Finland resulted in poorer outcomes for the exposed children later in life, not because gender was causally connected to student outcomes, but (in my view) because performance-related attributes were not identically distributed within male and female applicant pools.

3. For a detailed description and defense of holistic review see Bowen and Bok (2000).