Results

Data from Jakob et al. (2019). Collabra: Psychology, 5, 31. The authors investigated how the Hogwarts’ sorting hat test relates to some empirically established personality traits (in this example, Machiavellianism).

ANOVA

First, let's see if there are differences in Machiavellianism between the different houses using a one-way ANOVA.

As you can see in the output below, there is a significant difference in Machiavellianism between the different houses, F(3, 843) = 36.26, p < .001  (95% CI: 0.07, 0.15). Still, this does not tell us how any specific house scores on Machiavellianism compared to the others.

ANOVA - Machiavellianism
95% CI for ω²
Cases Sum of Squares df Mean Square F p ω² Lower Upper
Sorting house 3935.101 3 1311.700 36.259 < .001 0.111 0.073 0.150
Residuals 30496.517 843 36.176  
Note.  Type III Sum of Squares

Descriptives

The raincloud and descriptives plots below seem to suggest that Slytherin does score higher on Machiavellianism, and we can do a formal hypothesis test for such differences using either post hoc tests (where we conduct all pairwise comparisons) or contrast analysis (where we test specific, planned comparisons).


Descriptives plots

Raincloud plots


Machiavellianism

Contrast Tables

For the contrast analysis we have different options. We can either specify three custom contrasts, where each contrast compares Slytherin to one other house, but we can also specify the weights in such a way that we have a single comparison between Slytherin on the one hand, and the other three houses combined on the other hand. Because it's just so much fun to specify custom contrasts (just make sure that the weights always sum to 0), I've done both! The results indicate that Slytherin scores significantly higher on Machiavellianism than the other three houses combined (t(843) = -9.67 , p < .001, d = -2.65), but also when compared to each house individually. For all comparisons, the effect size is quite large (and with a narrow confidence interval thanks to the large sample size).


Custom Contrast - Sorting house
95% CI for Mean Difference 95% CI for Cohen's d
Comparison Estimate Lower Upper SE df t p Cohen's d Lower Upper
1 -15.923 -19.157 -12.690 1.647 843 -9.666 < .001 -2.647 -3.200 -2.095
2 -4.698 -5.915 -3.482 0.620 843 -7.582 < .001 -0.781 -0.987 -0.576
3 -4.569 -5.836 -3.303 0.645 843 -7.083 < .001 -0.760 -0.973 -0.546
4 -6.656 -7.923 -5.388 0.646 843 -10.307 < .001 -1.107 -1.324 -0.889
Custom Contrast Coefficients - Sorting house
Sorting house Comparison 1 Comparison 2 Comparison 3 Comparison 4
Ravenclaw 1 1 0 0
Gryffindor 1 0 1 0
Slytherin -3 -1 -1 -1
Hufflepuff 1 0 0 1

Post Hoc Tests

If we want to analyse all pairwise differences, a post hoc test (with Tukey's p-value correction) is more suitable. Based on the output below, we can see how each sorting house differs from the others. Focusing on Slytherin, we can see that it scores significantly higher on Machiavellianism compared to Ravenclaw t(843) = -7.58, p < .001, compared to Gryffindor t(843) = -7.08, p < .001, and compared to Hufflepuff t(843) = 10.31, p < .001 (note that the sign of the t-value changes depending on the reference point in the table, and how the t-statistics are identical to those from the contrast analysis). Because we look at all differences, we also see that Hufflepuff significantly scores lower than the other houses - how adorable!


Standard (HSD)

Post Hoc Comparisons - Sorting house
95% CI for Mean Difference
Mean Difference Lower Upper SE df t ptukey
Ravenclaw Gryffindor -0.129 -1.542 1.284 0.549 843 -0.235 0.995
  Slytherin -4.698 -6.294 -3.103 0.620 843 -7.582 < .001 ***
  Hufflepuff 1.957 0.543 3.372 0.550 843 3.562 0.002 **
Gryffindor Slytherin -4.569 -6.230 -2.909 0.645 843 -7.083 < .001 ***
  Hufflepuff 2.086 0.598 3.574 0.578 843 3.609 0.002 **
Slytherin Hufflepuff 6.656 4.993 8.318 0.646 843 10.307 < .001 ***
 ** p < .01, *** p < .001
Note.  P-value and confidence intervals adjusted for comparing a family of 4 estimates (confidence intervals corrected using the tukey method).