The NCAA Selection Procedure for 2000-2001

© 1998-2001, Joe Schlobotnik (archives)

URL for this frameset: http://slack.net/~whelan/tbrw/tbrw.cgi?2001/pairwise.shtml

See also

• 2001 NCAA Ice Hockey Championships Handbook
• You Are The Committee, an interactive run through the selection process (also available in a versatile self-service hypothetical interface)
• The "What If" interface to examine the selection criteria based on hypothetical outcomes of the conference tournaments, available in bracket or non-table forms
• The Season Ended Today, 2001 March 18
• How the 1998 seeding was determined
• NCAA seeds 1996-1998
• A Comparison of College Hockey Ranking Schemes
• A Proposed Modification to the NCAA Selection Criteria

Preamble

NCAA Men's Division I Ice Hockey is in a period of flux. In the past few years, two new leagues containing over a dozen newly tournament-eligible teams have joined the Division I family. One thing that has remained unchanged is that each year's championship tournament is selected and to some extent seeded entirely according to statistical analysis. As opposed to other sports, where subjective opinion polls and individual assessment of intangible factors invite allegations of bias (unintentional or otherwise), we college hockey fans have the luxury of knowing that our sport's tournament selection will proceed mostly by the numbers, and thus have a rough idea what the tournament field will look like before the selection committee even meets.

There have been some major changes to the procedure from past years, so we've largely re-written our description of the procedure. We've tried to put changes from previous years' procedures in bold.

The basics

First of all, from the NCAA's point of view, only official games played between established Division I programs count towards the selection process. This season, those teams are

The remaining two CHA members--Findlay and Wayne State--are currently in transition to Division I hockey and are not eligible for the 2001 tournament, nor will games against them be used in the selection process.

Pairwise Comparisons

The underlying principle behind the current selection process is the pairwise comparison. One team is compared to another team based on five criteria (which were not changed for the 2000-2001 season):

Ratings Percentage Index

Also known as RPI, not to be confused with Rensselaer Polytechnic Institute. This is a weighted average measuring a team's record and strength of schedule and consisting of 0.35 times the team's winning percentage, plus 0.50 times their opponents' average winning percentage, plus 0.15 times their opponents' opponents' winning percentage. (the fine print)

Record in last 16 games

This was changed as of the 1999 tournament; previously the criterion was record in last 20 games.

Record vs. Teams Under Consideration

A Team Under Consideration, or TUC, is a team with a record of .500 or above in games that count towards the selection process, as defined above. (the fine print)

Record vs. Common Opponents

This criterion compares the overall winning percentage of the two teams in question against all (eligible) opponents who appear on both their schedules, regardless of how many times they each play those teams. (the fine print)

Head-to-head record

A team wins one point towards the comparison for each of the first four criteria, and one point for each head-to-head game in which they defeated the other team in the comparison. Whichever team gets more points wins the comparison, and if it's a tie, the team with the higher RPI wins.

Every Team Under Consideration is compared to every other TUC in this way. The total number of such comparisons won is called the Pairwise Rating (PWR--the fine print). This number can be used to rank the TUCs, and in the past it was believed that the teams were seeded in the order of these Pairwise Rankings, but that is not precisely how it's done. The PWR is used to get a rough sense of which teams are in contention for which spots, but then those teams are placed according to the pairwise comparisons among or between them. For example, if you're battling it out for the twelfth and final spot in the postseason, it doesn't matter how you compare with the fifth-rated team. Thus a two-way tie is impossible, since one team will always win the pairwise comparison. If three teams end up in an unresolvable tie (rock-scissors-paper), we go to the RPI to resolve the deadlock.

One wild card: Conference strengths

In recent years, the deluge of new Division I programs, and the formation of new conferences in which those teams play the lion's share of their games against one another, have brought to light some of the weaknesses of the RPI and other selection criteria. Two-time MAAC regular-season champion Quinnipiac finished the past two seasons ranked in the top 12 in the national RPI rankings and held a pairwise comparison advantage over all but 9 teams each year, but were not included in the NCAA's field of 12. This was presumably related to the following paragraph in the NCAA News report on the Summer 1998 Division I Men's Ice Hockey Committee meeting:

In addition to revising one of its selection criteria, the committee noted that it reserves the right to evaluate each team based on the relative strength of their respective conference using the overall conference ratings percentage index (RPI) in determining competitive equity.

We still don't know what measure was used to determine this lack of competitive equity, but with the MAAC losing all nine of its games against members of the four established conferences in the 1999-2000 season, and accumulating a winning percentage below .300 against the rest of Division I, it was presumably a pretty easy call:

Less obvious was the technique used to evaluate Niagara's performance last season. In keeping with the selection criteria, the Purple Eagles were admitted to the NCAA field of 12, but they were seeded below Boston College and Michigan State, despite winning pairwise comparisons with each of them. So the committee apparently paid some attention to Niagara's performance in the selection criteria, but fudged their seeds downward somewhat to take into account their weaker College Hockey America schedule.

At any rate, the reason for Quinnipiac's (and Niagara's) deceptively high RPI and PWR is no big mystery. RPI attempts to correct a team's winning percentage for their strength of schedule by mixing it with the average winning percentage of their opponents. However, if those opponents have also played abnormally weak schedules, their winning percentages will be a poor indicator of their strength, and hence of the schedule strength of the team in question. According to the more sophisticated (the fine print) KRACH rating system, Quinnipiac was rated #41 out of 52 teams in 1999 and #44 out of 54 in 2000. The pairwise comparison algorithm is even more fragile, as the "Last 16" and "Teams Under Consideration" criteria make no allowance for strength of schedule at all, simply comparing the teams' winning percentages in those games. Niagara, despite having a low RPI, was able to win a few key comparisons last year by accumulating good records against weak teams in their last 16 games and against teams which accumulated winning records against weak schedules.

A modification to the selection critieria has been proposed which addresses these problems, but the NCAA has opted to stick with the present criteria and let the committee subjectively downgrade teams from weaker conferences.

The bottom line is that the committee is at liberty to leave CHA and MAAC teams out of the tournament, or grant them lower seeds, on the basis of the relative weakness of their schedules, even if their pairwise comparisons would otherwise entitle them to a berth or a better seed. (Unfortunately, this method cannot correct for the other consequences of RPI's shortcomings, such as the potential overvaluing of top MAAC and CHA opponents appearing on major conference teams' schedules this season.) Here is a table of each conference's average RPI and their record vs each other conference; additionally, the team with the best RPI in the conference is listed as well as the average RPI of their conference opponents.

For comparison, here is how the KRACH rating system predicts each conference would fare if each of its teams played each Division I team in each other conference once.

What do we do with the numbers?

The NCAA tournament consists of twelve teams, divided for the first round and a half into two regionals, East and West. In each regional, two teams receive first-round byes while the other four play on the first night. On the second night of the regional, the two bye teams play the two first-round winners, with the two survivors from each regional then advancing to the national semifinals the following weekend. The selection and seeding process can be divided into the following steps:

Qualification

The champions of five of the six Division I conferences (the WCHA, CCHA, ECAC, MAAC and Hockey East) receive automatic berths. Each of the five conferences has chosen to designate the winner of the conference tournament as the champion. (Note that the regular season champion is no longer guaranteed a berth, nor does College Hockey America receive any automatic berths.) The remaining seven spots in the tournament are at-large berths.

This is one of the places where our understanding of the process is still a little lacking. We know that the committee gives "obvious" at-large bids to teams that win comparisons with the rest of the candidates, then scrutinizes the "bubble" teams by comparing them individually to one another. Usually, the precise mechanics of this process are irrelevant, but in the 1999 selections, there were between two and four conceivable sets of tournament teams depending on how the bubble was pared down. We know that Ohio State and Northern Michigan got the last two bids in that particular season, but there were a couple of different lines of reasoning that could have given that result, and the selection committee hasn't explained which one was used.

Byes

The top four teams in the country, according to pairwise comparisons, are granted one- or two-seeds, and thus first-round byes in the regionals. Which region a team comes from is irrelevant. (I.e., three of the bye teams, or in principle even all four, can come from the same region.)

Regions

In dividing the twelve-team field into two six-team regionals, there is one absolute: the two regional hosts (Boston University in the East and Western Michigan in the West) play in their own regionals if they qualify for the tournament.

The top two teams in the nation (according to pairwise comparisons) automatically receive 1-seeds in the two regionals; the #1 team plays in its own region, the #2 team in the other region. This means that if the top two teams in the nation are from the same region, the #2 team is "shipped out" to the other regional. It hasn't been made explicit what happens to the #3 and #4 teams, but it seems reasonable to presume that if they come from different regions, each plays in their own regional.

There are now four remaining spots in each regional to fill with the other eight teams. The standard for the NCAA, if the tournament field consists of six teams from each region, is to "ship out" two teams from each region to play in the opposite regional. With the reduction in the numer of automatic bids, it is now theoretically possible (although unlikely) to have tournament fields which are regionally very unbalanced. It's not known what the NCAA would do, for instance with ten Eastern and two Western teams. In practice, 7/5 splits are not uncommon, and in such cases the committee has sent either two or three teams from the "overpopulated" region to play in the other regional, and swapped one or two teams out of the "underpopulated" region. All things being equal, these swapped teams should be the lowest-ranked in their respective regions according to pairwise comparisons. But considerations such as attendance and conference affilitation are taken into account, as described in "Fine Tuning" below.

Seedings and Pairings

Once the four non-bye teams in each regional are determined, they are placed in the three to six positions according to their pairwise comparisons. The four and five seeds will play in the first round, with the winner to face the one seed, while the three and six seeds will meet for the right to play the two seed.

Fine Tuning

At this point, we have a setup for the tournament according to the numbers, but there could be other problems with it. For instance, all four first-round contests could be rematches of the conference title games, or the teams with the biggest fan bases could be playing outside of their regions. These are both considered undesirable by the NCAA, so the committee can shuffle things a bit, either by altering the seedings within a region, or choosing to send different teams to the opposite regionals. First-round intra-conference matchups are positively verboten, and potential second-round games between teams in the same conference should be avoided, especially if the teams met in their conference playoffs. If two teams are swapped within a region to eliminate a second-round matchup, the other two teams will be swapped as well to retain the first-round pairings, if that doesn't cause more problems. Also, teams can be shifted to different regions to increase attendance.

This is the one part of the selection procedure which is really a judgement call on the committee's part, and thus the most unpredictable. Ordering teams within a regional is basically deterministic, but when deciding which non-bye teams go in which regional, the committee is supposed to consider

• pairwise comparisons
• attendance considerations
• avoidance of intraconference matchups

How much weight they give to each is completely unspecified, although attendance seems to be very important, while conference considerations are not a big priority in populating the regionals. The best way to guess what they'll do has been to look at historical precedent. But with this year's changes to the guidelines, it remains to be seen just how the committee will put them into practice.

Putting it in practice

If you want to see how this breaks down step-by-step with the current results (updated daily from the USCHO Division I Composite Schedule), you can now use the "You Are The Committee" tournament selection script, which takes you interactively through the process.