Analysis of PISA 2006 Preferred Items Ranking Using the Percent-Correct Method

This paper uses an approximate average percent-correct methodology to compare the ranks that wouldbe obtained for PISA 2006 countries if the rankings had been derived from items judged by each country tobe of highest priority for inclusion. The results reported show a remarkable consistency in the country rankorderings across different sets of countries’ preferred items when comparing with the rank reported in thePISA 2006 initial report (OECD, 2007). On average, only few countries systemically go up or down intheir ranking position. As these countries are in a group of moderate performers with very comparableoutcomes, these shifts in the ranking would probably be associated with minor changes in meanperformance on the final PISA scale. The analysis suggests that PISA rankings are noticeably stable thanksto the large enough pool of test items able to accommodate diverse preferences. The paper shows howimportant it is to base a choice of test items on a properly structured process which allows different expertsand countries to equally contribute. The evidence presented demonstrates that in PISA, average rankpositions of countries across different sets of preferred items are apparently stable and experts are not ableto predict which items can elevate performance of their countries in the final test.