Two years ago, after North Dakota was left out of the NCAA tournament instead of Dartmouth and Minnesota-Duluth was placed at Wisconsin first round, a longtime women’s college hockey expert wrote an e-mail to college hockey coaches explaining why the Pairwise Rankings — more specifically the RPI component — doesn’t work for the sport.
The e-mail was written by Harvard alum David DeRemer, who is probably the best expert on the Pairwise Rankings and the NCAA selection process I have ever come across — men’s or women’s side (DeRemer correctly predicted UND would get in this year over Wisconsin, despite the readings on the USCHO Pairwise).
Below is an excerpt of his e-mail to NCAA coaches in 2011:
Despite its widespread adoption by NCAA committees, the ratings percentage index (RPI) is a deeply flawed system with a bias towards interconference parity. To see why the RPI is flawed, consider a world with two conference where every team in conference A is better than every team in conference B, and each team only plays one nonconference game. As the number of conference games grows large, the RPIs of teams in each conference would converge to each team’s conference record, even though every team in conference A is better. While I don’t claim every team in the WCHA is better than every team in Hockey East, this is a rough approximation of why BC and BU ended up ahead of Minnesota, despite most human voters and statistically sound rankings placing Minnesota higher.
Statistics professors with interests in college hockey have developed multiple ranking systems which do not suffer from this bias towards parity in the RPI. One is the KRACH, hosted by USCHO. Another is the Rutter rankings. Both rankings use methods well accepted by the statisticians. Both rankings come to the conclusion that Minnesota is No. 3, UMD is much better than No. 7, and UND should have been No. 7. I’ll briefly describe each rating. For any pair of teams i and j with ratings A_i and A_j, the KRACH assumes the probability of team i winning is (A_i / (A_i + A_j)). The KRACH then simply picks the ratings so that the probabilities best match the actual results. Rutter does much the same, except it uses a slightly different formula for the probability involving the normal distribution and uses a slightly different estimation method.
When the RPI is flawed, it spills over into other criteria as well. For example, both KRACH and Rutter support the conclusion that Bemidji is one of the top 12 teams. Since record vs. top 12 teams is a selection criteria, WCHA teams were denied appropriate credit for their results against Bemidji.
Women’s hockey has better reasons than any other NCAA sport to institute a better system than the RPI and should take the institutional lead in doing so. Few sports other than women’s hockey have had a conference as successful as the WCHA that plays such a small share of their schedules against the other large conferences, yet the sport still attempts to conduct a truly national tournament. The RPI parity bias problem I described is stronger in women’s hockey than in most other sports.
It is important that WCHA members and interested parties and media respond to this injustice not by personally attacking the integrity of committee members or by suggesting minor tweaks like putting greater priority on “protecting the top seed” or “avoiding intraconference matchups.” The fundamental problem with the current bracket is that the NCAA selection criteria unfairly underrate all WCHA teams. If the committee had used a better criterion than the RPI, the pairing of two of the nation’s best teams, who have combined for the last 5 NCAA titles and met in 3 consecutive Frozen Fours, would have been avoided.
Now, if women’s college hockey were using the Rutter Rankings, which are mentioned in the e-mail, not only would UND be in the tournament, it would be a top-four seed hosting a regional this weekend. Also, Wisconsin would be in the tournament under the other rankings (I think we all know the Badgers were a top-8 team this year, too).
3. North Dakota
4. Boston University
6. Boston College
3. Boston University
4. Boston College
5. North Dakota
2. North Dakota
5. Boston College
6. Boston University