Last Modified: 2011 October 9
Joe Schlobotnik / joe@amurgsval.org
URL for this frameset: http://slack.net/~whelan/tbrw/tbrw.cgi?2002/pairwise.shtml
The story of the selection and seeding of teams for the NCAA Men's Division I Ice Hockey tournament is one of good news and bad news. The good news is that each year's tournament is selected and to some extent seeded entirely according to objective statistical analysis. As opposed to other sports, where subjective opinion polls and individual assessment of intangible factors invite allegations of bias (unintentional or otherwise), we college hockey fans have the luxury of knowing that our sport's tournament selection will proceed mostly by the numbers, and thus have a rough idea what the tournament field will look like before the selection committee even meets.
The bad news is that we've become increasingly uncertain about the details of this process over the past few years, as the addition of two new leagues containing over a dozen newly tournament-eligible teams has strained the effectiveness of the selection criteria and allowed a hint subjectivity to come back into the process in the form of some license for the committee to overrule the numbers in the case of widely varying conference strength.
First of all, from the NCAA's point of view, only official games played between established Division I programs count towards the selection process. This season, those teams are
The remaining CHA member, Findlay, is currently in transition to Division I hockey and not eligible for the 2002 tournament, nor will games against them be used in the selection process.
The underlying principle behind the current selection process is the pairwise comparison. One team is compared to another team based on five criteria:
A team wins one point towards the comparison for each of the first four criteria, and one point for each head-to-head game in which they defeated the other team in the comparison. Whichever team gets more points wins the comparison, and if it's a tie, the team with the higher RPI wins.
Every Team Under Consideration is compared to every other TUC in this way. The total number of such comparisons won is called the Pairwise Rating (PWR--the fine print). This number can be used to rank the TUCs, and in the past it was believed that the teams were seeded in the order of these Pairwise Rankings, but that is not precisely how it's done. The PWR is used to get a rough sense of which teams are in contention for which spots, but then those teams are placed according to the pairwise comparisons among or between them. For example, if you're battling it out for the twelfth and final spot in the postseason, it doesn't matter how you compare with the fifth-rated team. Thus a two-way tie is impossible, since one team will always win the pairwise comparison. If three teams end up in an unresolvable tie (rock-scissors-paper), we go to the RPI to resolve the deadlock.
In recent years, the deluge of new Division I programs, and the formation of new conferences in which those teams play the lion's share of their games against one another, have brought to light some of the weaknesses of the RPI and other selection criteria. Two-time MAAC regular-season champion Quinnipiac finished their first two Division I seasons ranked in the top 12 in the national RPI rankings and held a pairwise comparison advantage over all but 9 teams each year, but were not included in the NCAA's field of 12. This was presumably related to the following paragraph in the NCAA News report on the Summer 1998 Division I Men's Ice Hockey Committee meeting:
In addition to revising one of its selection criteria, the committee noted that it reserves the right to evaluate each team based on the relative strength of their respective conference using the overall conference ratings percentage index (RPI) in determining competitive equity.
There's no codified standard for competitive equity, but in its first two seasons the MAAC's out-of-conference performance made it a no-brainer to exclude Quinnipiac, and last season Mercyhurst received the MAAC's first auto-bid, so the distinction was not necessary.
Less obvious was the technique used to evaluate Niagara's performance in 1999-2000. In keeping with the selection criteria, the Purple Eagles were admitted to the NCAA field of 12, but they were seeded below Boston College and Michigan State, despite winning pairwise comparisons with each of them. So the committee apparently paid some attention to Niagara's performance in the selection criteria, but fudged their seeds downward somewhat to take into account their weaker College Hockey America schedule.
At any rate, the reason for Quinnipiac's (and Niagara's) deceptively high RPI and PWR is no big mystery. RPI attempts to correct a team's winning percentage for their strength of schedule by mixing it with the average winning percentage of their opponents. However, if those opponents have also played abnormally weak schedules, their winning percentages will be a poor indicator of their strength, and hence of the schedule strength of the team in question. According to the more sophisticated (the fine print) KRACH rating system, Quinnipiac was rated #41 out of 52 teams in 1999 and #44 out of 54 in 2000. The pairwise comparison algorithm is even more fragile, as the "Last 16" and "Teams Under Consideration" criteria make no allowance for strength of schedule at all, simply comparing the teams' winning percentages in those games. Niagara, despite having a low RPI, was able to win a few key comparisons last year by accumulating good records against weak teams in their last 16 games and against teams which accumulated winning records against weak schedules.
A modification to the selection critieria has been proposed which addresses these problems, but the NCAA has opted to stick with the present criteria and let the committee subjectively downgrade teams from weaker conferences.
The bottom line is that the committee is at liberty to leave CHA and MAAC teams out of the tournament, or grant them lower seeds, on the basis of the relative weakness of their schedules, even if their pairwise comparisons would otherwise entitle them to a berth or a better seed. (Unfortunately, this method cannot correct for the other consequences of RPI's shortcomings, such as the potential overvaluing of top MAAC and CHA opponents appearing on major conference teams' schedules this season.)
Here is a table of each conference's average RPI and their record vs each other conference; additionally, the team with the best RPI in the conference is listed as well as the average RPI of their conference opponents.
| Conference | Avg RPI | vs HE | vs WCHA | vs CCHA | vs CHA | vs ECAC | vs MAAC | Leader | Opp RPI |
|---|---|---|---|---|---|---|---|---|---|
| Hockey East (H) | .5362 | 5-12-1 | 8-7-2 | 4-2 | 32-9-1 | 9-0-2 | NH | .5252 | |
| WCHA (W) | .5360 | 12-5-1 | 10-11-1 | 16-4 | 13-2 | 0-0 | DU | .5260 | |
| CCHA (C) | .5192 | 7-8-2 | 11-10-1 | 12-3-1 | 14-8-3 | 6-0 | MS | .5124 | |
| CHA (A) | .4762 | 2-4 | 4-16 | 3-12-1 | 2-4-1 | 19-2 | WS | .4681 | |
| ECAC (E) | .4717 | 9-32-1 | 2-13 | 8-14-3 | 4-2-1 | 10-1-1 | Cr | .4619 | |
| MAAC (M) | .4354 | 0-9-2 | 0-0 | 0-6 | 2-19 | 1-10-1 | Mh | .4247 |
For comparison, here is how the KRACH rating system predicts each conference would fare if each of its teams played each Division I team in each other conference once.
| Conference | vs WCHA | vs Hockey East | vs CCHA | vs ECAC | vs CHA | vs MAAC |
|---|---|---|---|---|---|---|
| WCHA | .6237 | .6370 | .8278 | .8699 | .9829 | |
| Hockey East | .3763 | .5109 | .7342 | .7882 | .9680 | |
| CCHA | .3630 | .4891 | .7309 | .7865 | .9680 | |
| ECAC | .1722 | .2658 | .2691 | .5654 | .9089 | |
| CHA | .1301 | .2118 | .2135 | .4346 | .8972 | |
| MAAC | .0171 | .0320 | .0320 | .0911 | .1028 |
The NCAA tournament consists of twelve teams, divided for the first round and a half into two regionals, East and West. In each regional, two teams receive first-round byes while the other four play on the first night. On the second night of the regional, the two bye teams play the two first-round winners, with the two survivors from each regional then advancing to the national semifinals the following weekend. The selection and seeding process can be divided into the following steps:
The champions of five of the six Division I conferences (the WCHA, CCHA, ECAC, MAAC and Hockey East) receive automatic berths. Each of the five conferences has chosen to designate the winner of the conference tournament as the champion. (Note that the regular season champion is not guaranteed a berth, nor does College Hockey America receive any automatic berths.) The remaining seven spots in the tournament are at-large berths.
This is one of the places where our understanding of the process is still a little lacking. We know that the committee gives "obvious" at-large bids to teams that win comparisons with the rest of the candidates, then scrutinizes the "bubble" teams by comparing them individually to one another. Usually, the precise mechanics of this process are irrelevant, but in the 1999 selections, there were between two and four conceivable sets of tournament teams depending on how the bubble was pared down. We know that Ohio State and Northern Michigan got the last two bids in that particular season, but there were a couple of different lines of reasoning that could have given that result, and the selection committee hasn't explained which one was used.
The top four teams in the country, according to pairwise comparisons, are granted one- or two-seeds, and thus first-round byes in the regionals. Which region a team comes from is irrelevant. (I.e., three of the bye teams, or in principle even all four, can come from the same region.)
In dividing the twelve-team field into two six-team regionals, there is one absolute: the two regional hosts (Boston University in the East and Michigan in the West) play in their own regionals if they qualify for the tournament.
Presumably, the top team in the country according to pairwise comparisons will be the #1 seed in their own region. The remaining three by teams need to be placed so that the fourth-best team overall is bracket to play the top team overall in the semis, which means they need to be the #2 seed in the other region. All else being equal, the second-best team in the country will also be kept in their own region, either as the #2 seed behind the national #1 or the #1 seed in the other region.
There are now four remaining spots in each regional to fill with the other eight teams. In past years, the NCAA has "shipped out" two teams from each region to play in the opposite regional. However, in aftermath of the terrorist attacks on September 11, 2001, the NCAA laid down some guidelines to reduce travel last fall and recently extended them to cover winter and spring sports. The bottom line is that aside from seeding the top four correctly and avoiding first-round intraconference matchups, the committee will place teams in regionals so as to minimize air travel. This will generally mean keeping them in their own region, although in some cases (e.g., Mercyhurst) the opposite regional may be closer, while other teams (e.g., Denver and Colorado College) may have to fly to either regional.
Once the four non-bye teams in each regional are determined, they are placed in the three to six positions according to their pairwise comparisons. The four and five seeds will play in the first round, with the winner to face the one seed, while the three and six seeds will meet for the right to play the two seed.
Given the extra considerations of reducing air travel, there will probably be less fine-tuning than in past years. (E.g., fewer teams will be playing outside of their regions, so attendance is less likely to be an issue.) But efforts to avoid first-round games between teams in the same conference may lead to altering the seeding within a regional.
At any rate, these sorts of adjustments are the one part of the selection procedure which is really a judgement call on the committee's part, and thus the most unpredictable. Within the constraints of reducing air travel, the committee can also consider the following factors:
How much weight they give to each is completely unspecified, although attendance seems to be very important, while conference considerations are not a big priority in populating the regionals. The best way to guess what they'll do has been to look at historical precedent. But reverse-engineering the process is tricky, especially given the extra guidelines in place this year.
If you want to see how this breaks down step-by-step with the current results (updated daily from the USCHO Division I Composite Schedule), you can now use the "You Are The Committee" tournament selection script, which takes you interactively through the process. And if you want to see my take on the process using some recent results, have a look at the latest "If the Season Ended Today" column.