Why The Pairwise Doesn’t Work For Women’s Hockey

Two years ago, after North Dakota was left out of the NCAA tournament instead of Dartmouth and Minnesota-Duluth was placed at Wisconsin first round, a longtime women’s college hockey expert wrote an e-mail to college hockey coaches explaining why the Pairwise Rankings — more specifically the RPI component — doesn’t work for the sport.

The e-mail was written by Harvard alum David DeRemer, who is probably the best expert on the Pairwise Rankings and the NCAA selection process I have ever come across — men’s or women’s side (DeRemer correctly predicted UND would get in this year over Wisconsin, despite the readings on the USCHO Pairwise).

Below is an excerpt of his e-mail to NCAA coaches in 2011:

Despite its widespread adoption by NCAA committees, the ratings percentage index (RPI) is a deeply flawed system with a bias towards interconference parity. To see why the RPI is flawed, consider a world with two conference where every team in conference A is better than every team in conference B, and each team only plays one nonconference game. As the number of conference games grows large, the RPIs of teams in each conference would converge to each team’s conference record, even though every team in conference A is better. While I don’t claim every team in the WCHA is better than every team in Hockey East, this is a rough approximation of why BC and BU ended up ahead of Minnesota, despite most human voters and statistically sound rankings placing Minnesota higher.

Statistics professors with interests in college hockey have developed multiple ranking systems which do not suffer from this bias towards parity in the RPI. One is the KRACH, hosted by USCHO. Another is the Rutter rankings. Both rankings use methods well accepted by the statisticians. Both rankings come to the conclusion that Minnesota is No. 3, UMD is much better than No. 7, and UND should have been No. 7. I’ll briefly describe each rating. For any pair of teams i and j with ratings A_i and A_j, the KRACH assumes the probability of team i winning is (A_i / (A_i + A_j)). The KRACH then simply picks the ratings so that the probabilities best match the actual results. Rutter does much the same, except it uses a slightly different formula for the probability involving the normal distribution and uses a slightly different estimation method.

When the RPI is flawed, it spills over into other criteria as well. For example, both KRACH and Rutter support the conclusion that Bemidji is one of the top 12 teams. Since record vs. top 12 teams is a selection criteria, WCHA teams were denied appropriate credit for their results against Bemidji.

Women’s hockey has better reasons than any other NCAA sport to institute a better system than the RPI and should take the institutional lead in doing so. Few sports other than women’s hockey have had a conference as successful as the WCHA that plays such a small share of their schedules against the other large conferences, yet the sport still attempts to conduct a truly national tournament. The RPI parity bias problem I described is stronger in women’s hockey than in most other sports.

It is important that WCHA members and interested parties and media respond to this injustice not by personally attacking the integrity of committee members or by suggesting minor tweaks like putting greater priority on “protecting the top seed” or “avoiding intraconference matchups.” The fundamental problem with the current bracket is that the NCAA selection criteria unfairly underrate all WCHA teams. If the committee had used a better criterion than the RPI, the pairing of two of the nation’s best teams, who have combined for the last 5 NCAA titles and met in 3 consecutive Frozen Fours, would have been avoided. 

Now, if women’s college hockey were using the Rutter Rankings, which are mentioned in the e-mail, not only would UND be in the tournament, it would be a top-four seed hosting a regional this weekend. Also, Wisconsin would be in the tournament under the other rankings (I think we all know the Badgers were a top-8 team this year, too).

Rutter Rankings

1. Minnesota
2. Cornell
3. North Dakota
4. Boston University
5. Wisconsin
6. Boston College
7. Harvard
8. Clarkson

KRACH rankings

1. Minnesota
2. Cornell
3. Boston University
4. Boston College
5. North Dakota
6. Harvard
7. Wisconsin
8. Clarkson

Massey Rankings

1. Minnesota
2. North Dakota
3. Cornell
4. Wisconsin
5. Boston College
6. Boston University
7. Harvard
8. Clarkson

11 Responses

  1. One thing seems certain for the WCHA, if the NCAA is not going to overhaul their system for placement in the Tournament, then the WCHA had better protect its brand.

    As was suggested by someone else, the most obvious way is to reduce the number of Conference games played so they may play some of the Eastern teams more, and in theory, garner more wins and improve their RPI. Though I disagree with the play 3 times scenario. Best way might be to reduce Conference schedule from 28 games to 22 or 24 and rotate the teams you play twice vs. four times. Similar to what the Men did when they expanded to 12 teams.

    Interesting note that the top 7 Strength of Schedules all belong to the 8 team WCHA! At least prior to last weekend, that was true. Have not checked this week.

  2. Ben

    Could not agree more. But unfortunately, I think the powers that be are more than willing to let the current system continue. Only way to force a change is to either (1) play more NC games or (2) have MN or UND absolutely dominate the Frozen Four winning by rediculous margins.

    The real championship game is this weekend between UND & MN.

    1. Goldy

      The Gophers have not lost in more than a year. This is one of those siutaitons where somebody will be bittersweet-delighted to win second place.

      The season is long and I know they are more fired up to win the D1 crown (on home ice) than they were to end the regualr season vs. UND. …The next game will likely not be very close. UND should be proud of their 2-0 loss as it will be their crowning achievement in the 2012-13 caompaign!

      The studettes the Gophers have coming in will likley go another two with less than a handful of losses

      1. Sara

        Arrogant much? See this is where this entitlement crap coming from the mighty U comes from.

        A 2-0 loss will be our crowning achievement? Give me a break.

        1. GoldyII

          Honestly, it will be as close as anybody comes to knocking them off this year. Have you watched any of the games?

    2. Siouxguy

      UND could get beat by any of the teams remaining, so to say the championship is this weekend, seems off base. After all, UND did lose to Clarkson at home, which isn’t ranked as high as some of the other teams in the tournament. True, UND is playing better, but who is to say some of the other teams aren’t as well.

  3. Paul

    At a very simple level the other three rankings agree 100% on the 8 teams in that should be in the tournament albeit in different orders. The only ranking system that doesn’t agree on the 8 teams is the one that is in use today.

    1. Skater88

      Of particular note is that one school that makes it every year(via the current system), is not listed in any of these rankings. This is what gives value to those rankings. The fact that a team with a schedule like Mercyhurst gets a bid without question does a severe injustice to the sports other teams in contention.

  4. Dave

    This is a situation where the college presidents are asleep at the wheel. The NCAA knows the RPI is flawed but they aren’t going to do anything about it because they always go where the money is. The college presidents need to step up and demand better treatment for their teams and student athletes.

Comments are closed.