A Proposed Modification to the NCAA Selection Criteria

© 2000, Joe Schlobotnik (archives)

System developed with Ken Butler, Gary Hatfield and Bob Stagat

URL for this frameset: http://slack.net/~whelan/tbrw/tbrw.cgi?2000/kpairwise.3.shtml

See Also

• The NCAA Selection Procedure
• A Comparison of College Hockey Ranking Schemes
• KRACH rating system
• The Season Ended Today, 2000 March 19
• "You Are The Committee" interactive tournament selection script

Motivation

Definitions of Proposed Modifications

Effects of Proposed Modifications

The actual pairwise comparisons for the 1999-2000 season using the NCAA's standard selection criteria can be found either on my Rankings Comparison Page or on US College Hockey Online's Pairwise Rankings Page. The table below summarizes the pairwise comparisons that the modified scheme would have produced. It lists the teams in order of total number of comparisons won (KPWR), with ties broken by individual comparisons between teams, or failing that by comparing teams' KRACH ratings, which are shown in the table in the form of the expected winning percentage (RRWP) for a team if they were to play a full round robin schedule, one game against each other eligible team. In each team's row are the abbreviations of all the teams they defeat in our Bradley-Terry modified pairwise comparisons. Clicking on these abbreviations will call up a popup window containing a breakdown of the five criteria for that pair of teams. Clicking on a team's name will open a new window containing a list of all their opponents in order of KRACH rating, their records against each opponent, and in the subset of those games applying to the "vs Teams Under Consideration" and "Last 16 Games" criteria.

Tournament Selection

According to the modified criteria, twelve teams (including the seven who received automatic bids for winning conference championships) win pairwise comparisons with all the other teams under consideration and would therefore have made up the twelve-team tournament field. Eleven of these twelve were in the actual NCAA field this year; the twelfth is Minnesota State-Mankato, who would have replaced Niagara. The ranking of the teams according to pairwise comparisons would have been Wisconsin, North Dakota, St. Cloud State, Michigan State, Michigan, and Mankato in the West and Maine, New Hampshire, Boston College, St. Lawrence, Boston University and Colgate in the East.

Note that Quinnipiac, who win the standard pairwise comparisons with St. Cloud as well as every team not in the tournament this year, win comparisons only with other MAAC teams in the modified system. They could therefore have been excluded on the basis of the comparisons alone, with no reference to the overall strengths of the various conferences.

Reversed Comparisons

Of the 325 comparisons among the 26 teams under consideration this season, 61 would have been won by the other team if the proposed modifications to the criteria had been in effect. Of those, 13 involve Quinnipiac, 10 Niagara, and 8 involve MAAC teams other than Quinnipiac. Of the 66 comparisons among the 12 teams in this year's tournament, 15 would have gone the other way under the modified system; 6 of those involved Niagara.

The following table details each comparison which has different results in the standard and modified systems. These are grouped by the team winning the modified comparison. For each of the first four criteria (RPI/KRACH, games vs TUCs, last 16 games, and common opponents) a W, L or T shows respectively a win for the first team, a win for the second team, or a tie in that criterion, first in the modified system and then in the standard one. The fifth criterion is head-to-head results, with each head-to-head game counting the same as each of the other criteria. A '+' or '-' number indicates that the team winning the modified comparison has that many wins more or fewer, respectively, in head-to-head competition.

Next we will examine closely several of the pairwise comparisons reversed under the proposed system which would have had a substantial impact on the selection and seeding of this year's tournament. To paint strength of schedule in broad strokes, it is useful to divide teams into four groups based on their KRACH ratings:

300 and above (12 teams)

Wisconsin, NoDak, Maine, UNH, BC, BU, SLU, SCSU, Michigan, Colgate, Michigan State, Minnesota

100-300 (19 teams)

Mankato, CC, RPI, PC, NMU, Niagara, FSU, UAA, Clarkson, Northeastern, Cornell, LSSU, Merrimack, DU, UMD, UNO, Notre Dame, Mass-Amherst, BGSU

34-100 (11 teams)

Miami, UVM, Harvard, Princeton, OSU, WMU, Mass-Lowell, Dartmouth, Yale, Brown, Union

33 and under (12 teams)

UAF, Quinnipiac, MTU, Air Force, Canisius, Iona, Sacred Heart, Army, UConn, Holy Cross, AIC, Fairfield

A KRACH of 100 corresponds to a RRWP of .500, i.e., a team which would be expected to win half of its games if it played every other eligible team an equal number of times. Such a team would be expected to go .000 against a team with an infinite KRACH, .250 against one with a KRACH of 300, .500 against another team with a 100 rating, .750 against a team with a rating of 33.3, and 1.000 against a team with a KRACH of 0.

The first group of teams (300 and up) includes all this year's tournament participants except Niagara, as well as Minnesota. Of the 26 teams with .500 or better records, all but the four in the MAAC are among the 31 teams in the first two groups (100 and up). The lowest group (under 33.3) includes all eight eligible MAAC teams, both service academies, and bottom-feeders Alaska-Fairbanks (5-25-2 overall) and Michigan Tech (4-34).

St. Cloud vs Quinnipiac

All of the comparisons which Quinnipiac wins with non-MAAC teams would have been lost using the modified criteria. Included among those are comparisons with Michigan, Michigan State and St. Cloud State, all of whom made the NCAAs. The first two received automatic bids, but St. Cloud was granted an at-large bid in lieu of Quinnipiac, on the grounds that the MAAC was a weak conference and inflated Quinnipiac's performance in the selection criteria. Here is how the comparison looks in each system

Quinnipiac and SCSU have no head-to-head games and no common opponents, but the remaining criteria are a clean sweep for the Braves. They have a higher Ratings Percentage Index and a better record in their last 16 games and against Teams Under Consideration. But in the modified system, SCSU is the clear winner according to all three criteria; their performance in each case is ten times as good as Quinnipiac's.

The RPI consists 35% of a team's winning percentage and 65% of their strength of schedule as measured by ten parts opponents' winning percentage and 3 parts opponents' opponents' winning percentage. Since Quinnipiac's winning percentage is .160 higher than St. Cloud's (.768 to .608), SCSU's strength of schedule would need to be .086 better to make up the difference in the RPI. Instead, it's only .0555 better (.5124 to .4569). However, Quinnipiac's strength of schedule is much weaker than RPI's measure implies. Of their 28 games, 24 were against other MAAC teams, all of whom were in the lowest of our four classes (KRACH of 33 or below), three against teams in the above average group (100-300) and only one against a team of the first class (300+). However, according to RPI's measure of strength, only AIC (.365), Holy Cross (.363) and Fairfield (.171), accounting for 11 games, are below-average opponents. In particular, Canisius (.616) helps Quinnipiac's strength of schedule more than Mankato (.589) helps St. Cloud's, and Sacred Heart (.581) is considered a stronger opponent than Minnesota (.571). KRACH, on the other hand, sees Quinnipiac has played and lost four games against teams with ratings of 150.9, 175.9, 194.9, and 615.7, and gone 20-1-3 against teams with ratings ranging from 7.986 to .4275 and places them in between the two groups at 30.43. (Note that a win in any of those four non-conference games would have approximately doubled Quinnipiac's KRACH.)

But even without the RPI, Quinnipiac would win this comparison on the basis of a better record against other TUCs and in their last 16 games. But while St. Cloud went 8-7-3 playing 14 games against TUCs of the highest caliber (KRACH 300+) and 4 more against respectable TUCs (100-300), Quinnipiac's 6-5-3 consisted of one game against a top-flight team (Maine), three more against above-average ones (RPI, Niagara, Clarkson), and 10 against teams at the bottom of the barrel (Canisius, Iona, Sacred Heart) who just happened to amass winning records against extremely weak schedules. Hence the criterion rating for Quinnipiac of only 20.15, while SCSU's was 515.5, indicating that their performance against TUCs was even better than their overall performance, when strength of schedule is taken into account. Similarly, Quinnipiac's last 16 games included Maine, Clarkson, and 14 MAAC games, so going 12-3-1 in those games is far less of an accomplishment than SCSU's 9-5-2 against only above-average teams.

It's easy to see, looking under the hood of this comparison, that Quinnipiac's win according to the standard criteria was meaningless, and equally easy for the committee to overrule those criteria. But with the modified criteria, the comparison would have gone against QC anyway, without need for special action.

Michigan State vs Niagara

Niagara University won pairwise comparisons with all but five teams under consideration, and while they played in a fairly weak conference (CHA, whose only other tournament-eligible members were Army and Air Force), they also had a number of non-conference games against respectable opposition. The committee granted them an at-large berth, but seeded them lower than their PWCs would have indicated. One of the comparisons which they apparently overruled was that with Michigan State:

Using the standard criteria, this is a clean sweep for Niagara, but all of the criteria besides common opponents swing the other way when strength of schedule is taken into account. Since RPI (KRACH in the modified system) is used as a tie-breaker, Michigan State needs to win RPI/KRACH plus at least one other criterion to overcome Niagara's advantage. First, a look at how Niagara is overrated according to RPI. Niagara's schedule included 3 games against teams at the top according to KRACH (300+), 7 against other above-average teams (100-300), 7 against teams somewhat below average (34-100), and 10 against the bottom of the bunch (33-). So if you take away the five wins over Air Force, Niagara would have gone 13-7-2 against a fairly balanced schedule, and KRACH values that at an above-average 175.9 (which translates to a projected winning percentage (RRWP) of .607 if they played every eligible team once). However, as far as RPI's strength of schedule measure is concerned, Air Force was a rather respectable opponent at .596. (Part of the problem here is that none of AFA's five losses to Niagara count when calculating their contribution to Niagara' strength of schedule.) Additionally, Canisius (.541) is considered an above-average opponent, and Quinnipiac is the toughest competition Niagara faced as far as RPI is concerned (.718). By those standards, the only opponent that strong on Michigan State's schedule was Wisconsin at .720.

One thing that stands out about Niagara's performance is the seeming disparity between some of their wins against strong to very respectable teams like Boston University and Colorado College and their losses to weak sisters like Canisius and Union. This has given them a reputation as a "big game team", and the NCAA tries to favor such teams come tournament time with the criterion of record vs other teams under consideration. And indeed, Niagara's criterion rating against teams under consideration is almost 2/3 higher than their overall KRACH. But when comparing their 5-3 record in these games to Michigan State's 8-5-3, strength of schedule again becomes important. Of Niagara's TUC games, 3 were against top teams (300+, and all in the tournament), 3 against above-average teams (100-300), and 2 were against very weak teams (Quinnipiac and Canisius). Michigan State, in contrast, played 6 games against top-notch TUCs and 6 more against respectable ones. When you consider that Quinnipiac and Canisius should have been gimmes for a team amassing a record around .500 against the better teams in the nation, and that Michigan State played several games above .500 against bona fide TUCs, it seems reasonable that Niagara's criterion rating of 294.3 is lower than MSU's 359.2. The last 16 games tell a similar story, with half of Niagara's being played against Air Force and the MAAC, while only Michigan State's two against Alaska-Fairbanks fall into the "very weak" category.

Minnesota State-Mankato vs Niagara

This was probably the most important comparison in the whole selection procedure this year; Mankato was next in line for an at-large berth if Niagara had been left out of the tournament. In awarding the bid to Niagara, the committee in effect indicated that while the Purple Eagles' comparison wins over Boston College and Michigan State were not valid given relative conference strengths, the win over Mankato was. So let's have a look:

Just as in the comparison with Michigan State, Niagara wins all four criteria in the present system, but loses all but common opponents in the modified one. Starting again with the Ratings Percentage Index, we have already noted that Air Force, Quinnipiac and Canisius are viewed by the RPI as disproportionately strong opponents. Comparing the "strengths" (10 parts winning percentage and 3 parts opponents' winning percentage) for Niagara's and Mankato's opponents, we see that Quinnipiac is considered a stronger opponent than North Dakota, Air Force stronger than St. Cloud, and Canisius stronger than Colorado College. The "strength of schedule" contributions to the RPI for the two teams are almost the same; Niagara's is .5015 and Mankato's .5011. Thus, as far as RPI is concerned, the two teams played equally strong schedules, and Niagara is higher rated because of their .704 to .603 advantage in winning percentage. The idea that Mankato's schedule was no stronger than Niagara's is a pretty nonsensical one. Of the Mavericks' 35 games, 11 were against teams with KRACH ratings above 300, 20 against teams rated between 100 and 300, none against teams with ratings between 34 and 100, and four against a team rated below 33 (Michigan Tech, rated at 25.71). We've already broken down Niagara's schedule; the numbers are 3-7-7-10. So rather than both teams' opposition being about average, Mankato's was topheavy, while Niagara's was slightly weak. When their respective records against the differing schedules are considered by KRACH, it is Mankato who come out ahead. In fact, if each team had played all of the other Division I teams once each, it is Mankato who would have come out with the .695 winning percentage (instead of .603) and Niagara's would have been .607 (instead of .704).

Record against Teams Under Consideration is another criterion in which Niagara seems to have the advantage at first, 5-3 to 7-8-2. But as we've noted before, two of the TUCs Niagara played were Quinnipiac and Canisius, and beating them is hardly an achievement on par with knocking off a Clarkson or North Dakota. Aside from those two games, Niagara went .500 against a balanced assortment of TUCs (3 games each above and below the 300 KRACH mark). MSU-Mankato's record was slightly below .500, with 11 of the 17 games against teams with KRACH ratings above 300 and the remaining 6 against teams with KRACHs between 100 and 300. So, a schedule somewhat tougher than Niagara's (leaving out the Quinnipiac and Canisius games) and a slightly worse record. Understandable, then, that the modified criterion favors Mankato by a small margin.

Record in the last 16 games, on the other hand, is not at all close when strength of schedule is taken into account. Niagara went 12-2-2 to MSU-Mankato's 10-4-2, but their schedule in those games included no games against top-notch teams (300+), 4 against above-average teams (100-300), 4 more against somewhat below-average teams (34-100), and the remaining 8 against the weaklings (33 and below). Meanwhile, Mankato's breakdown in their last 16 games was 6-8-0-2. In other words, while Niagara was spending half their homestretch beating up on the likes of Air Force, Mankato was playing almost exclusively above-average teams, and still managed to play six games above .500. It's easy to see that strength of schedule will tip this comparison to the Mavericks, and in fact their criterion rating is over twice Niagara's.

So the modified criteria give the pairwise comparison to MSU-Mankato, who have a higher KRACH and better criterion ratings for games against TUCs and the last 16 games, while Niagara maintains their advantage against common opponents. Overall, against TUCs, and in the last 16 games, Mankato accumulated a lower winning percentage against a tougher schedule than Niagara. However, the latter two criteria in the NCAA's present system do not consider strength of schedule, while the RPI uses a measure of strength of schedule which considers Mankato and Niagara to have played equally tough opposition; this can be attributed to the fact that several of Niagara's opponents (Quinnipiac, Air Force, Canisius) compiled relatively high winning percentages against weak schedules.

Comparing the Michigan State-Niagara comparison, which the selection committee overruled, and the Mankato-Niagara one, which they did not, in each case the modified comparison is reversed thanks to the KRACH, vs TUC, and last 16 criteria, but only one of the last two, along with KRACH, is necessary. In both cases an equally obvious case can be made for Niagara's last 16 advantage being due not to an inherently better performance, but due to a much weaker schedule in those games; at a glance:

So the key question is the validity of Niagara's RPI relative to Michigan State and Minnesota State-Mankato. Here is a table summarizing once more the important data:

According to RPI, Michigan State accumulated more or less the same winning percentage as Niagara against a slightly weaker schedule, while Mankato State amassed a somewhat lower winning percentage than Niagara against a schedule of almost the same strength. Looking carefully at Niagara's opponents, we see that their schedule is in fact substantially weaker than the other two, and RPI is blinded to that fact by the overrating of Quinnipiac, Air Force, and Canisius. So Michigan State, achieving nearly as good a winning percentage as Niagara, against a tougher schedule, naturally has a higher KRACH. The question is whether Mankato's lower winning percentage is made up for by their stronger schedule or not. KRACH says it is, and that's why they win the modified pairwise comparison.

New Hampshire vs Colgate

The proposed modifications to the selection criteria would also have an impact on comparisons between members of the four established Division I conferences. The decision of which Eastern teams to send West might have been affected by the reversal of Colgate's comparisons with Boston College and New Hampshire; the decision of which Western teams to send East may have been affected by the reversal of Michigan and Michigan State's comparisons with St. Cloud; and finally, the seeding of the East Regional might have been affected by the reversal of Colgate's comparison with Michigan. The latter comparisons hinge on the RPI/KRACH criterion, so let's instead focus on the simpler case of Colgate's comparisons in the East. Looking at the table of reversed comparisons above, we see that in both systems, UNH wins the RPI/KRACH criterion and Colgate the Last 16 and Common Opponents ones, so the difference in the "vs TUCs" criterion. Here's the usual table breaking that down:

New Hampshire's record is only slightly worse, and the TUCs they've played are significantly stronger than Colgate's (the Red Raiders' strongest opponent is was St. Lawrence, and UNH played 11 games against teams with higher KRACH ratings than SLU). The end result is that UNH's criterion rating is somewhat higher than Colgate's (although not overwhelmingly so).

Boston College vs Colgate

This comparison is a lot more obvious, since it hinges on the last 16 games, with BC having the higher RPI and KRACH, and Colgate doing better against TUCs and common opponents, with or without consideration of strength of schedule. The rundown of each team's last 16 games:

In this case, the won-lost-tied record is only slightly in Colgate's favor (a mere two points in 16 games), while the caliber of BC's recent opponents was clearly a lot higher. Only one of Colgate's last 16 games was against a team who made the tournament, while 8 of BC's were. This large imbalance leads to a much higher criterion rating for BC. Colgate's recent performance was impressive, but if they had played a schedule like BC's we'd have to conclude by extrapolation they'd have managed significantly less than 24 points in those 16 games.

A 16-Team Tournament

There has been talk that the NCAA tournament will likely be expanded to 16 teams next season, at the same time as the MAAC becomes eligible for an automatic bid for their conference champion in their third year of existence. It's interesting to consider who would have made such a tournament this season, if the modified system were in effect. In the table above, we see 16 teams who all win modified PWCs with all the other teams under consideration: the 12 participants in this year's tournament, plus Mankato, Minnesota, Providence and Colorado College. However, with an automatic bid going to MAAC playoff champion Connecticut, one at-large team would be displaced, and the team losing comparisons (in principle, the addition of UConn as a team under consideration could change the outcome of the "vs TUC" criterion, but none of the teams involved played the Huskies this season) with all the rest is Niagara. (The CHA is a year behind the MAAC, so won't qualify for an auto-bid next season.) So in a 16-team tournament field chosen according to the modified criteria with a MAAC auto-bid, the at-large bids would have gone to UNH, BC, SCSU, Colgate, Mankato, Minnesota, Providence, and CC.

The Fate of Niagara

It may seem a little harsh to say that under this system, Niagara would not receive a bid even to a 16-team tournament. However, this is directly related to the outcomes of Niagara's games. The Purple Eagles were upset by Canisius in the closing weeks of the season. Using the self-service hypothetical interface to my tournament selection script, one can calculate the modified pairwise comparisons if Niagara had won that game, and see that this would have been enough to pull out the comparisons with Colorado College and Providence, causing Niagara to replace Providence in a hypothetical 16-team field.

A month earlier, Niagara played two-game series at ECAC cellar dweller Brown and took three points. If, in addition to avoiding the Canisius upset, Niagara had completed their sweep of Brown, they would have also ended up winning their modified comparisons with Colgate and Minnesota State-Mankato (and Michigan State, for that matter); that would have been enough for Niagara to warrant an at-large bid in place of Mankato in the modified system as well as the standard one. Similar results would have been obtained if the Purple Eagles had defeated Canisius and taken four rather than three points from Nebraska-Omaha at the end of the season.

So according to the proposed modified system, Niagara's fate would have hinged on the results of their last three regular season games, which were against a MAAC team and a team that finished the season with a losing record. Three wins would have ultimately been enough to earn an at-large bid. The fact that they ended up going 1-1-1 in those games is what led to their falling short to Mankato and the others in the modified criteria. The same is true of teams who fall just short of bids every year. An extra win here or there makes the difference between "close but no cigar" and a ticket to the dance.