Trends in PISA science scores

I noticed from the PISA results that the Science scores appear to be decreasing, since 2006.

(OECD 2023b, 471)

I wanted to know whether this trend was statistically significant. So I set about completing an anova test. This was using PISA data from 2009, 2015, 2018, and 2022.

First, I checked that the data was normally distributed. For all four data sets this was shown to be true.

 UKSCI2009 <-PISA_2009 %>% 
  select(CNT, PV1SCIE) %>% 
  filter(CNT == "GBR")%>% 
  mutate(year ="2009")   

UKSCI2015 <-PISA_2015 %>% 
  select(CNT, PV1SCIE) %>% 
  filter(CNT == "GBR") %>% 
  mutate(year ="2015") 

UKSCI2018 <-PISA_2018 %>% 
  select(CNT, PV1SCIE) %>% 
  filter(CNT == "United Kingdom")%>% 
  mutate(year ="2018") 

UKSCI2022 <-PISA_2022 %>% 
  select(CNT, PV1SCIE) %>% 
  filter(CNT == "United Kingdom")%>% 
  mutate(year ="2022")  
 
TotalPISA <- rbind(UKSCI2022, UKSCI2018, UKSCI2015, UKSCI2009)

ggplot(data = TotalPISA, aes(x=PV1SCIE, fill = year)) +
  geom_histogram(bins=100) +
  facet_wrap(.~year)

ggplot(data = TotalPISA, aes(sample = PV1SCIE, colour = year)) +
  geom_qq() +
  geom_qq_line(color = "black", linetype = "dashed") +
  facet_wrap(.~year, ncol =2)

I then checked the variances of each of the pairs of data sets.

VarM <- var(UKSCI2022$PV1SCIE, na.rm = TRUE) 

VarF <- var(UKSCI2018$PV1SCIE, na.rm = TRUE) 

VarM / VarF

[1] 1.162336

VarM <- var(UKSCI2022$PV1SCIE, na.rm = TRUE) 

VarF <- var(UKSCI2015$PV1SCIE, na.rm = TRUE) 

VarM / VarF

[1] 1.117499

VarM <- var(UKSCI2022$PV1SCIE, na.rm = TRUE) 

VarF <- var(UKSCI2009$PV1SCIE, na.rm = TRUE) 

VarM / VarF

[1] 1.083176

The variance ratio was close to 1, for all pairs, so the two conditions were met and so I could then perform the anova test.

The null hypothesis would be that the mean of PV1SCIE for all the years would be the same, with the alternative hypothesis being statically different.

Since I had 4 sets of PISA data, this meant that I wanted to perform multiple t-tests, so therefore I completed an anova test.

Anova test

result <- aov(data =TotalPISA, PV1SCIE ~ year) 
sumanova <- summary(result) 
sumanova

               Df    Sum Sq Mean Sq F value Pr(>F)    
year            3   2511892  837297    87.4 <2e-16 ***
Residuals   53122 508920819    9580                   
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The Pr(>F) value is <2e-16 which is over 0.05. This suggests that the PV1SCIE values are statistically different.

My question is: Why is it that there is a difference in the PV1SCIE scores. Why are they going down? What are the factors that could be causing this?

To further look at the results I ran a Tukey Test following on from the Anova. A Tukey Test can be used to determine which years have significant differences, in this case the mean PV1SCIE scores in the UK.

TukeyHSD(result)

  Tukey multiple comparisons of means
    95% family-wise confidence level

Fit: aov(formula = PV1SCIE ~ year, data = TotalPISA)

$year
                diff        lwr          upr     p adj
2015-2009  -6.602382  -9.710095  -3.49466982 0.0000003
2018-2009 -14.879938 -18.005230 -11.75464592 0.0000000
2022-2009 -17.860524 -21.033200 -14.68784778 0.0000000
2018-2015  -8.277555 -11.284561  -5.27054992 0.0000000
2022-2015 -11.258141 -14.314365  -8.20191730 0.0000000
2022-2018  -2.980586  -6.054684   0.09351177 0.0613144

Interestingly, we cannot say there is a statistical difference between 2022 and 2018, since the p value is larger than 0.05. All other comparisons are statistically difference. Therefore I would agree with OECD (OECD 2023a) that the decline of Science values are not all down to the pandemic. There must be other factors for the decrease in science values. It would be interesting to investigate what is causing this pattern.

References

OECD. 2023a. “Decline in Educational Performance Only Partly Attributable to the COVID-19 Pandemic.” OECD. https://www.oecd.org/newsroom/decline-in-educational-performance-only-partly-attributable-to-the-covid-19-pandemic.htm.

———. 2023b. “PISA 2022 Results (Volume i): The State of Learning and Equity in Education.” OECD. https://doi.org/10.1787/53f23881-en.