The Judgment of Princeton
At its Annual Conference in Princeton, the American Association of Wine Economists organized a wine tasting called “The Judgment of Princeton.” It was modeled after the 1976 “Judgment of Paris.” In 1976, British wine merchant Steve Spurrier organized two blind tastings with 9 French wine judges who were associated with the wine industry in various ways (wine journalists, critics, sommeliers, merchants or winemakers). In the first flight, the judges rated 10 white wines, 6 from Napa and 4 from Burgundy. In the second flight, the judges rated 10 reds, 6 from Napa and 4 from Bordeaux, France. In both tastings a wine from Napa, a then relatively unknown wine region, was declared the winner. George Taber of TIME magazine, the only attending journalist, reported the results to the world causing a big surprise in France, and helping to put Napa wines on the global wine map.
At the Princeton tasting, led by George Taber, 9 wine judges from France, Belgium and the U.S. tasted French against New Jersey wines. The French wines selected were from the same producers as in 1976 including names such as Chateau Mouton-Rothschild and Haut Brion, priced up to $650/bottle. New Jersey wines for the competition were submitted to an informal panel of judges, who then selected the wines for the competition. These judges were not eligible to taste wines at the final competition The results were similarly surprising. Although, the winner in each category was a French wine (Clos de Mouches for the whites and Mouton-Rothschild for the reds) NJ wines are at eye level. Three of the top four whites were from New Jersey. The best NJ red was ranked place 3. An amazing result given that the prices for NJ average at only 5% of the top French wines.
A statistical evaluation of the tasting, conducted by Princeton Professor Richard Quandt, further shows that the rank order of the wines was mostly insignificant. That is, if the wine judges repeated the tasting, the results would most likely be different. From a statistically viewpoint, most wines were undistinguishable. Only the best white and the lowest ranked red were significantly different from the others wines.
In Paris, after the identity of the wines was revealed, Odette Kahn, editor of La Revue Du Vin De France, demanded her score card back. Apparently, she was not happy with having rated American wines number one and two.
At the Princeton blind tasting, both French judges preferred NJ red wines over their counterparts from Bordeaux. After disclosing the wines’ identity the French judges were surprised but did not complain.
FULL TEXT PDF
WINETASTER ON 06/08/12 WITH 9 JUDGES AND 10 WINES BASED ON GRADES, IDENT=N
Copyright (c) 1995-2012 Richard E. Quandt, V. 1.65
FLIGHT 1:
Number of Judges = 9
Number of Wines = 10
Identification of the Wine: The judges' overall ranking:
Wine A is Heritage Chardonnay 2010 ....... 3rd place
Wine B is Unionville Pheasant Hill Single Vineyard 2010 ........ 2nd place
Wine C is Puligny Montrachet Domaine Leflaive 2009 ........ 5th place
Wine D is Clos des Mouches Drouhin 2009 ........ 1st place
Wine E is Silver Decoy "Black Feather" 201 ........ 4th place
Wine F is Bellview Chardonnay 2010 tied for 6th place
Wine G is Ventimiglia Chardonnay 2010 ........ 9th place
Wine H is Meursault-Charmes Jean Latour-Labille2008 ........ 10th place
Wine I is Amalthea Chardonnay 2008 ........ 8th place
Wine J is Bâtard Montrachet Marc-Antonin Blain 2009 tied for 6th place
The Judges' Grades
Judge Wine -> A B C D E F G H I J
Jean-M Cardebat 10.0 13.0 14.0 15.0 8.0 13.0 15.0 11.0 9.0 12.0
Tyler Colman 16.0 14.0 14.0 16.0 12.0 11.0 11.0 14.0 11.0 14.0
John Foy 16.0 17.0 16.0 15.0 14.5 14.5 16.0 17.0 15.0 17.5
Olivier Gergaud 14.0 19.0 12.0 10.0 19.0 18.0 17.0 16.0 18.0 14.0
Robert Hodgson 17.0 11.0 13.0 14.0 14.0 10.0 9.0 9.0 10.0 10.0
Linda Murphy 15.5 15.0 17.0 18.0 16.0 17.0 15.0 14.0 16.0 17.0
Daniele Meulder 10.0 15.0 12.0 12.0 15.0 14.0 15.0 12.0 15.0 12.0
Jamal Rayyis 16.0 15.0 14.5 17.5 16.5 14.0 12.0 15.0 13.0 12.0
Francis Schott 17.0 16.0 12.0 18.0 15.0 16.0 15.0 14.0 17.0 15.0
The Judges' Rankings
Judge Wine -> A B C D E F G H I J
Jean-M Cardebat 8.0 4.5 3.0 1.5 10.0 4.5 1.5 7.0 9.0 6.0
Tyler Colman 1.5 4.5 4.5 1.5 7.0 9.0 9.0 4.5 9.0 4.5
John Foy 5.0 2.5 5.0 7.5 9.5 9.5 5.0 2.5 7.5 1.0
Olivier Gergaud 7.5 1.5 9.0 10.0 1.5 3.5 5.0 6.0 3.5 7.5
Robert Hodgson 1.0 5.0 4.0 2.5 2.5 7.0 9.5 9.5 7.0 7.0
Linda Murphy 7.0 8.5 3.0 1.0 5.5 3.0 8.5 10.0 5.5 3.0
Daniele Meulder 10.0 2.5 7.5 7.5 2.5 5.0 2.5 7.5 2.5 7.5
Jamal Rayyis 3.0 4.5 6.0 1.0 2.0 7.0 9.5 4.5 8.0 9.5
Francis Schott 2.5 4.5 10.0 1.0 7.0 4.5 7.0 9.0 2.5 7.0
Group Ranking -> 3 2 5 1 4 6 9 10 8 6
Votes Against -> 45.5 38.0 52.0 33.5 47.5 53.0 57.5 60.5 54.5 53.0
( 9 is the best possible, 90 is the worst)
Here is a measure of the correlation in the preferences of the judges which
ranges between 1.0 (perfect correlation) and 0.0 (no correlation):
W = 0.1017
The probability that random chance could be responsible for this correlation
is rather large, 0.5476. Most analysts would say that unless this
probability is less than 0.1, the judges' preferences are not strongly
related.
We now analyze how each taster's preferences are correlated with the group
preference. A correlation of 1.0 means that the taster's preferences are a
perfect predictor of the group's preferences. A 0.0 means no correlation,
while a -1.0 means that the taster has the reverse ranking of the group.
This is measured by the correlation R.
Correlation Between the Ranks of
Each Person With the Average Ranking of Others
Name of Person Correlation R
Robert Hodgson 0.7299
Jamal Rayyis 0.5321
Tyler Colman 0.2750
Francis Schott 0.2037
Linda Murphy 0.0502
Jean-M Cardebat -0.1774
Olivier Gergaud -0.3303
Daniele Meulders -0.3729
John Foy -0.4318
We now compute for each wine the mean (average) grade it received from the judges
and the standard deviation of those grades. (The mean of, say, 3 grades is
the sum of the grades divided by 3. The standard deviation is a measure of
the "dispersion" of the grades around the mean. The numbers 5, 6, and 7 and the
numbers 3, 6, and 9 both have the same mean of 6, but the second set has a
higher dispersion, i.e., strandard deviation.
Summary Grade Statistics for Wines
A B C D E F G H I J
Mean 14.61 15.00 13.83 15.06 14.44 14.17 13.89 13.56 13.78 13.72
StdDev 2.60 2.16 1.70 2.59 2.90 2.47 2.47 2.36 3.01 2.35
We now compute the mean and the standard deviation for each judge, from which
you can see which judge is harsh and which is lenient, which thinks that the
wines are very different and which thinks they are pretty much the same.
Summary Grade Statistics for Judges
Name Mean Std.Dev.
Jean-M Cardebat 12.00 2.32
Tyler Colman 13.30 1.85
John Foy 15.85 1.03
Olivier Gergaud 15.70 2.93
Robert Hodgson 11.70 2.53
Linda Murphy 16.05 1.15
Daniele Meulders 13.20 1.72
Jamal Rayyis 14.55 1.75
Francis Schott 15.50 1.63
The wines were preferred by the judges in the following order. When the
preferences of the judges are strong enough to permit meaningful differentiation
among the wines, they are separated by -------------------- and are judged to be
significantly different.
1. ........ 1st place Wine D is Clos des Mouches 2009
-----------------------------------------------------------------------
2. ........ 2nd place Wine B is Unionville Single Vineyard 2010
3. ........ 3rd place Wine A is Heritage Chard 2010
4. ........ 4th place Wine E is Silver Decoy "Black Feather" 201
5. ........ 5th place Wine C is Puligny Montrachet 2009
6. tied for 6th place Wine F is Bellview Chard 2010
7. tied for 6th place Wine J is Batard Montrachet 2009
8. ........ 8th place Wine I is Amalthea Chard 2008
9. ........ 9th place Wine G is Ventimiglia Chard 2010
10. ........ 10th place Wine H is Meursault-Charmes 2008
We now undertake a more detailed examination of the pair-wise rank correla-
tions that exist between pairs of judges. First, we present a table in which you
can find the correlation for any pair of judges, by finding one of the names in the
left hand margin and the other name on top of a column. A second table arranges
these correlations in descending order and marks which is significantly positive
significantly negative, or not significant. This may allow you to find clusters
of judges whose rankings were particularly similar or particularly dissimilar.
Pairwise Rank Correlations
Correlations must exceed in absolute value 0.65 for significance at the 0.05
level and must exceed 0.56 for significance at the 0.1 level
Jean-M Cardebat Tyler Colman John Foy
Jean-M Cardebat 1.000 0.115 0.140
Tyler Colman 0.115 1.000 0.393
John Foy 0.140 0.393 1.000
Olivier Gergaud -0.455 -0.638 -0.250
Robert Hodgson -0.174 0.602 -0.319
Linda Murphy 0.261 0.146 -0.394
Daniele Meulders -0.081 -0.763 -0.314
Jamal Rayyis -0.153 0.617 -0.312
Francis Schott -0.031 0.194 -0.363
Olivier Gergaud Robert Hodgson Linda Murphy
Jean-M Cardebat -0.455 -0.174 0.261
Tyler Colman -0.638 0.602 0.146
John Foy -0.250 -0.319 -0.394
Olivier Gergaud 1.000 -0.209 -0.464
Robert Hodgson -0.209 1.000 0.396
Linda Murphy -0.464 0.396 1.000
Daniele Meulders 0.816 -0.329 -0.263
Jamal Rayyis -0.098 0.733 0.075
Francis Schott -0.006 0.409 0.258
Daniele Meulders Jamal Rayyis Francis Schott
Jean-M Cardebat -0.081 -0.153 -0.031
Tyler Colman -0.763 0.617 0.194
John Foy -0.314 -0.312 -0.363
Olivier Gergaud 0.816 -0.098 -0.006
Robert Hodgson -0.329 0.733 0.409
Linda Murphy -0.263 0.075 0.258
Daniele Meulders 1.000 -0.260 0.000
Jamal Rayyis -0.260 1.000 0.314
Francis Schott 0.000 0.314 1.000
Pairwise correlations in descending order
0.816 Olivier Gergaud and Daniele Meulders Significantly positive
0.733 Robert Hodgson and Jamal Rayyis Significantly positive
0.617 Tyler Colman and Jamal Rayyis Significantly positive
0.602 Tyler Colman and Robert Hodgson Significantly positive
0.409 Robert Hodgson and Francis Schott Not significant
0.396 Robert Hodgson and Linda Murphy Not significant
0.393 Tyler Colman and John Foy Not significant
0.314 Jamal Rayyis and Francis Schott Not significant
0.261 Jean-M Cardebat and Linda Murphy Not significant
0.258 Linda Murphy and Francis Schott Not significant
0.194 Tyler Colman and Francis Schott Not significant
0.146 Tyler Colman and Linda Murphy Not significant
0.140 Jean-M Cardebat and John Foy Not significant
0.115 Jean-M Cardebat and Tyler Colman Not significant
0.075 Linda Murphy and Jamal Rayyis Not significant
0.000 Daniele Meulders and Francis Schott Not significant
-0.006 Olivier Gergaud and Francis Schott Not significant
-0.031 Jean-M Cardebat and Francis Schott Not significant
-0.081 Jean-M Cardebat and Daniele Meulders Not significant
-0.098 Olivier Gergaud and Jamal Rayyis Not significant
-0.153 Jean-M Cardebat and Jamal Rayyis Not significant
-0.174 Jean-M Cardebat and Robert Hodgson Not significant
-0.209 Olivier Gergaud and Robert Hodgson Not significant
-0.250 John Foy and Olivier Gergaud Not significant
-0.260 Daniele Meulders and Jamal Rayyis Not significant
-0.263 Linda Murphy and Daniele Meulders Not significant
-0.312 John Foy and Jamal Rayyis Not significant
-0.314 John Foy and Daniele Meulders Not significant
-0.319 John Foy and Robert Hodgson Not significant
-0.329 Robert Hodgson and Daniele Meulders Not significant
-0.363 John Foy and Francis Schott Not significant
-0.394 John Foy and Linda Murphy Not significant
-0.455 Jean-M Cardebat and Olivier Gergaud Not significant
-0.464 Olivier Gergaud and Linda Murphy Not significant
-0.638 Tyler Colman and Olivier Gergaud Significantly negative
-0.763 Tyler Colman and Daniele Meulders Significantly negative
WINETASTER ON 06/08/12 WITH 9 JUDGES AND 10 WINES BASED ON GRADES, IDENT=N
Copyright (c) 1995-2012 Richard E. Quandt, V. 1.65
FLIGHT 2:
Number of Judges = 9
Number of Wines = 10
Identification of the Wine: The judges' overall ranking:
Wine A is Ch. Montrose 2004 ........ 4th place
Wine B is Ch. Mouton Rothschild 2004 ........ 1st place
Wine C is Silver Decoy Cab. Franc 2008 ........ 8th place
Wine D is Heritage Estate BDX 2010 ........ 3rd place
Wine E is Bellview Lumiere 2010 ........ 7th place
Wine F is Tomasello Oak Reserve 2007 ........ 5th place
Wine G is Ch. Leoville Las Cases 2004 ........ 6th place
Wine H is Amalthea Europa VI 2008 ........ 9th place
Wine I is Four JG's Cab Franc 2008 ........ 10th place
Wine J is Ch. Haut Brion 2004 ........ 2nd place
The Judges' Grades
Judge Wine -> A B C D E F G H I J
Jean-M Cardebat 15.0 11.0 12.0 16.0 14.0 11.0 14.5 13.0 10.0 14.5
Tyler Colman 14.0 11.0 16.0 12.0 14.0 13.0 14.0 12.0 13.0 11.0
John Foy 17.5 19.0 18.0 18.0 15.0 16.0 18.0 18.0 17.0 17.5
Olivier Gergaud 10.0 17.0 9.0 14.0 19.0 12.0 15.0 10.0 11.0 18.0
Robert Hodgson 13.0 17.0 13.0 16.0 12.0 15.0 10.0 12.0 8.0 11.0
Linda Murphy 13.0 14.0 17.0 16.0 15.0 17.0 14.0 15.5 13.0 18.0
Daniele Meulder 14.0 16.0 11.0 16.0 14.0 15.0 13.0 11.0 10.0 15.0
Jamal Rayyis 15.0 19.5 14.0 12.0 13.0 16.0 14.5 15.0 16.0 16.0
Francis Schott 19.0 18.0 8.0 15.0 15.0 12.0 15.0 16.0 7.0 17.0
The Judges' Rankings
Judge Wine -> A B C D E F G H I J
Jean-M Cardebat 2.0 8.5 7.0 1.0 5.0 8.5 3.5 6.0 10.0 3.5
Tyler Colman 3.0 9.5 1.0 7.5 3.0 5.5 3.0 7.5 5.5 9.5
John Foy 6.5 1.0 3.5 3.5 10.0 9.0 3.5 3.5 8.0 6.5
Olivier Gergaud 8.5 3.0 10.0 5.0 1.0 6.0 4.0 8.5 7.0 2.0
Robert Hodgson 4.5 1.0 4.5 2.0 6.5 3.0 9.0 6.5 10.0 8.0
Linda Murphy 9.5 7.5 2.5 4.0 6.0 2.5 7.5 5.0 9.5 1.0
Daniele Meulder 5.5 1.5 8.5 1.5 5.5 3.5 7.0 8.5 10.0 3.5
Jamal Rayyis 5.5 1.0 8.0 10.0 9.0 3.0 7.0 5.5 3.0 3.0
Francis Schott 1.0 2.0 9.0 6.0 6.0 8.0 6.0 4.0 10.0 3.0
Group Ranking -> 4 1 8 3 7 5 6 9 10 2
Votes Against -> 46.0 35.0 54.0 40.5 52.0 49.0 50.5 55.0 73.0 40.0
( 9 is the best possible, 90 is the worst)
Here is a measure of the correlation in the preferences of the judges which
ranges between 1.0 (perfect correlation) and 0.0 (no correlation):
W = 0.1543
The probability that random chance could be responsible for this correlation
is rather large, 0.2039. Most analysts would say that unless this
probability is less than 0.1, the judges' preferences are not strongly
related.
We now analyze how each taster's preferences are correlated with the group
preference. A correlation of 1.0 means that the taster's preferences are a
perfect predictor of the group's preferences. A 0.0 means no correlation,
while a -1.0 means that the taster has the reverse ranking of the group.
This is measured by the correlation R.
Correlation Between the Ranks of
Each Person With the Average Ranking of Others
Name of Person Correlation R
Daniele Meulders 0.7963
Francis Schott 0.3877
Robert Hodgson 0.3272
Jean-M Cardebat 0.2778
John Foy 0.1443
Olivier Gergaud 0.0823
Linda Murphy -0.1074
Jamal Rayyis -0.1296
Tyler Colman -0.6927
We now compute for each wine the mean (average) grade it received from the judges
and the standard deviation of those grades. (The mean of, say, 3 grades is
the sum of the grades divided by 3. The standard deviation is a measure of
the "dispersion" of the grades around the mean. The numbers 5, 6, and 7 and the
numbers 3, 6, and 9 both have the same mean of 6, but the second set has a
higher dispersion, i.e., strandard deviation.
Summary Grade Statistics for Wines
A B C D E F G H I J
Mean 14.50 15.83 13.11 15.00 14.56 14.11 14.22 13.61 11.67 15.33
StdDev 2.47 3.00 3.28 1.89 1.83 2.02 1.97 2.49 3.20 2.59
We now compute the mean and the standard deviation for each judge, from which
you can see which judge is harsh and which is lenient, which thinks that the
wines are very different and which thinks they are pretty much the same.
Summary Grade Statistics for Judges
Name Mean Std.Dev.
Jean-M Cardebat 13.10 1.91
Tyler Colman 13.00 1.48
John Foy 17.40 1.09
Olivier Gergaud 13.50 3.44
Robert Hodgson 12.70 2.61
Linda Murphy 15.25 1.66
Daniele Meulders 13.50 2.06
Jamal Rayyis 15.10 1.93
Francis Schott 14.20 3.82
The wines were preferred by the judges in the following order. When the
preferences of the judges are strong enough to permit meaningful differentiation
among the wines, they are separated by -------------------- and are judged to be
significantly different.
1. ........ 1st place Wine B is Ch. Mouton Rothschild 2004
2. ........ 2nd place Wine J is Ch. Haut Brion 2004
3. ........ 3rd place Wine D is Heritage Estate BDX 2010
4. ........ 4th place Wine A is Ch.Montrose 2004
5. ........ 5th place Wine F is Tomasello Oak Reserve 2007
6. ........ 6th place Wine G is Leoville Las Cases 2004
7. ........ 7th place Wine E is Bellview Lumiere 2010
8. ........ 8th place Wine C is Silver Decoy Cab. Frfanc 2008
9. ........ 9th place Wine H is Amalthea Europa 2008
---------------------------------------------------------------------
10. ........ 10th place Wine I is Four JG's Cab Franc 2008
We now undertake a more detailed examination of the pair-wise rank correla-
tions that exist between pairs of judges. First, we present a table in which you
can find the correlation for any pair of judges, by finding one of the names in the
left hand margin and the other name on top of a column. A second table arranges
these correlations in descending order and marks which is significantly positive
significantly negative, or not significant. This may allow you to find clusters
of judges whose rankings were particularly similar or particularly dissimilar.
Pairwise Rank Correlations
Correlations must exceed in absolute value 0.65 for significance at the 0.05
level and must exceed 0.56 for significance at the 0.1 level
Jean-M Cardebat Tyler Colman John Foy
Jean-M Cardebat 1.000 0.053 0.120
Tyler Colman 0.053 1.000 -0.279
John Foy 0.120 -0.279 1.000
Olivier Gergaud 0.159 -0.394 -0.189
Robert Hodgson 0.061 -0.215 0.363
Linda Murphy 0.095 -0.209 -0.022
Daniele Meulders 0.336 -0.542 0.168
Jamal Rayyis -0.601 -0.538 0.019
Francis Schott 0.481 -0.414 0.318
Olivier Gergaud Robert Hodgson Linda Murphy
Jean-M Cardebat 0.159 0.061 0.095
Tyler Colman -0.394 -0.215 -0.209
John Foy -0.189 0.363 -0.022
Olivier Gergaud 1.000 -0.055 0.095
Robert Hodgson -0.055 1.000 0.203
Linda Murphy 0.095 0.203 1.000
Daniele Meulders 0.554 0.710 0.291
Jamal Rayyis 0.074 -0.006 -0.121
Francis Schott 0.246 0.290 -0.155
Daniele Meulders Jamal Rayyis Francis Schott
Jean-M Cardebat 0.336 -0.601 0.481
Tyler Colman -0.542 -0.538 -0.414
John Foy 0.168 0.019 0.318
Olivier Gergaud 0.554 0.074 0.246
Robert Hodgson 0.710 -0.006 0.290
Linda Murphy 0.291 -0.121 -0.155
Daniele Meulders 1.000 0.106 0.484
Jamal Rayyis 0.106 1.000 0.224
Francis Schott 0.484 0.224 1.000
Pairwise correlations in descending order
0.710 Robert Hodgson and Daniele Meulders Significantly positive
0.554 Olivier Gergaud and Daniele Meulders Not significant
0.484 Daniele Meulders and Francis Schott Not significant
0.481 Jean-M Cardebat and Francis Schott Not significant
0.363 John Foy and Robert Hodgson Not significant
0.336 Jean-M Cardebat and Daniele Meulders Not significant
0.318 John Foy and Francis Schott Not significant
0.291 Linda Murphy and Daniele Meulders Not significant
0.290 Robert Hodgson and Francis Schott Not significant
0.246 Olivier Gergaud and Francis Schott Not significant
0.224 Jamal Rayyis and Francis Schott Not significant
0.203 Robert Hodgson and Linda Murphy Not significant
0.168 John Foy and Daniele Meulders Not significant
0.159 Jean-M Cardebat and Olivier Gergaud Not significant
0.120 Jean-M Cardebat and John Foy Not significant
0.106 Daniele Meulders and Jamal Rayyis Not significant
0.095 Jean-M Cardebat and Linda Murphy Not significant
0.095 Olivier Gergaud and Linda Murphy Not significant
0.074 Olivier Gergaud and Jamal Rayyis Not significant
0.061 Jean-M Cardebat and Robert Hodgson Not significant
0.053 Jean-M Cardebat and Tyler Colman Not significant
0.019 John Foy and Jamal Rayyis Not significant
-0.006 Robert Hodgson and Jamal Rayyis Not significant
-0.022 John Foy and Linda Murphy Not significant
-0.055 Olivier Gergaud and Robert Hodgson Not significant
-0.121 Linda Murphy and Jamal Rayyis Not significant
-0.155 Linda Murphy and Francis Schott Not significant
-0.189 John Foy and Olivier Gergaud Not significant
-0.209 Tyler Colman and Linda Murphy Not significant
-0.215 Tyler Colman and Robert Hodgson Not significant
-0.279 Tyler Colman and John Foy Not significant
-0.394 Tyler Colman and Olivier Gergaud Not significant
-0.414 Tyler Colman and Francis Schott Not significant
-0.538 Tyler Colman and Jamal Rayyis Not significant
-0.542 Tyler Colman and Daniele Meulders Not significant
-0.601 Jean-M Cardebat and Jamal Rayyis Significantly negative