Bugs: Mixed Design 2 (Between Subjects) x 2 (Within Subject) x 2 (Within Subject) ANOVA

Description:

This data set, "Bugs", provides the extent to which men and women want to kill arthropods that vary in freighteningness (low, high) and disgustingness (low, high). Each participant rates their attitudes towards all anthropods. Subset of the data reported by Ryan et al. (2013).

Variables:

  1. Gender - Participant's gender (Woman, Man).
  2. Lo D, Lo F - Desire to kill an artrhopod with low freighteningness and low disgustingness.
  3. Lo D, Hi F - Desire to kill an artrhopod with low freighteningness and high disgustingness.
  4. Hi D, Lo F - Desire to kill an artrhopod with high freighteningness and low disgustingness.
  5. Hi D, Hi F - Desire to kill an artrhopod with high freighteningness and high disgustingness.
  6. For a description of the remaining variables see Ryan et al. (2013).

The desire to kill an arthropod was indicated on a scale from 0 to 10.

This example JASP file demonstrates the use of mixed design ANOVA. We will test the relation between hostility towards insects and their disgustingness and frighteningness for males and females separately.

References:

Ryan, R. S., Wilde, M., & Crist, S. (2013). Compared to a small, supervised lab experiment, a large, unsupervised web-based experiment on a previously unknown effect has benefits that outweigh its potential costs. Computers in Human Behavior, 29, 1295-1301.

Repeated Measures ANOVA

To conduct the mixed design ANOVA, select the ANOVA menu and click on Repeated Measures ANOVA. Add Disgustingness (Low/High) and Frighteningness (Low/High) as Repeated Measures Factors (just start typing to change names of factors and levels) and drag the disgustingness ratings into the right Repeated Measures Cells. Drag Gender into the Between Subject Factors box. Select the Estimates of effect size option from the Additional Options menu to display effect sizes - since we have multiple predictors, be sure to select the partial version of the effect sizes.

Within Subjects Effects
Cases Sum of Squares df Mean Square F p ω²ₚ
Disgustingness 49.174 1 49.174 12.061 < .001 0.023
Disgustingness ✻ Gender 1.760 1 1.760 0.432 0.513 0.000
Residuals 346.552 85 4.077  
Frighteningness 138.075 1 138.075 32.122 < .001 0.064
Frighteningness ✻ Gender 5.523 1 5.523 1.285 0.260 6.263×10-4
Residuals 365.366 85 4.298  
Disgustingness ✻ Frighteningness 13.799 1 13.799 4.688 0.033 0.006
Disgustingness ✻ Frighteningness ✻ Gender 13.799 1 13.799 4.688 0.033 0.006
Residuals 250.177 85 2.943  
Note.  Type III Sum of Squares

The Within Subjects Effects table shows the results for the repeated measures factors. Disgustingness and Frighteningness both have a significant effect on Hostility (p < 0.001). There is also a significant two-way interaction between Disgustingness and Frighteningness, F(1,85) = 4.688, p = 0.033, and a significant three-way interaction between Disgustingness, Frighteningness, and Gender, F(1,85) = 4.688, p = 0.033. The effect size for the three-way interaction effect is quite small (i.e.,  = 0.006), while the effect sizes for the two significant main effects are also fairly small (i.e.,  = 0.023 and = 0.064).

Between Subjects Effects
Cases Sum of Squares df Mean Square F p ω²ₚ
Gender 18.346 1 18.346 0.996 0.321 0.000
Residuals 1565.879 85 18.422  
Note.  Type III Sum of Squares

The Between Subjects Effects table shows the results for the between-subjects factor Gender. There is no statistically significant main effect of Gender, F(1,85) = 0.996, p = 0.321. 

Descriptives

The plots below can be created in the Descriptive Plots menu by dragging Disgustingness to the horizontal axis, Frighteningness to Separate lines, and Gender to Separate plots. The plot shows the nature of the three-way interaction that was indicated by the hypothesis test: there seems to be no interaction effect between Frighteningness and Disgustingness for women (i.e., parallel lines), while there appears some interaction for men (i.e., non-parallel lines).

Descriptives plots

Gender: Woman
Gender: Man

Assumption Checks

Select Homogeneity tests from the Assumption Checks menu to test the assumption of equal variances between women and men on all 4 repeated measures (looking at the ratio of variances yields a similar conclusion). To assess normality, we can look at the Q-Q plot of residuals - most dots are aligned on the diagonal, with perhaps some non-normality in the tail-areas. Since the sample size is fairly big, this is no reason for concern though.

Test for Equality of Variances (Levene's)
  F df1 df2 p
Lo D, Lo F 0.453 1 85 0.503
Lo D, Hi F 0.598 1 85 0.441
Hi D, Lo F 0.527 1 85 0.470
Hi D, Hi F 0.042 1 85 0.839

Q-Q Plot

Post Hoc Tests

Post Hoc Comparisons - Disgustingness ✻ Frighteningness - Conditional on Disgustingness
Disgustingness Mean Difference SE df t pholm
Low Low High -1.759 0.273 85 -6.445 < .001
High   High -0.914 0.336 85 -2.720 0.008
Note.  Results are averaged over the levels of: Gender
Post Hoc Comparisons - Disgustingness ✻ Frighteningness - Conditional on Frighteningness
Frighteningness Mean Difference SE df t pholm
Low Low High -1.220 0.325 85 -3.757 < .001
High   High -0.375 0.276 85 -1.359 0.178
Note.  Results are averaged over the levels of: Gender
Post Hoc Comparisons - Gender ✻ Disgustingness - Conditional on Gender
Gender Mean Difference SE df t pholm
Woman Low High -0.647 0.265 85 -2.439 0.017
Man   High -0.948 0.375 85 -2.529 0.013
Note.  Results are averaged over the levels of: Frighteningness
Post Hoc Comparisons - Gender ✻ Disgustingness - Conditional on Disgustingness
Disgustingness Mean Difference SE df t pholm
Low Woman Man 0.638 0.528 85 1.208 0.230
High   Man 0.336 0.551 85 0.611 0.543
Note.  Results are averaged over the levels of: Frighteningness
Post Hoc Comparisons - Gender ✻ Frighteningness - Conditional on Gender
Gender Mean Difference SE df t pholm
Woman Low High -1.603 0.272 85 -5.890 < .001
Man   High -1.069 0.385 85 -2.777 0.007
Note.  Results are averaged over the levels of: Disgustingness
Post Hoc Comparisons - Gender ✻ Frighteningness - Conditional on Frighteningness
Frighteningness Mean Difference SE df t pholm
Low Woman Man 0.220 0.581 85 0.378 0.706
High   Man 0.754 0.500 85 1.510 0.135
Note.  Results are averaged over the levels of: Disgustingness

Copy of Repeated Measures ANOVA

To conduct the mixed design ANOVA, select the ANOVA menu and click on Repeated Measures ANOVA. Add Disgustingness (Low/High) and Frighteningness (Low/High) as Repeated Measures Factors (just start typing to change names of factors and levels) and drag the disgustingness ratings into the right Repeated Measures Cells. Drag Gender into the Between Subject Factors box. Select the Estimates of effect size option from the Additional Options menu to display effect sizes - since we have multiple predictors, be sure to select the partial version of the effect sizes.

Within Subjects Effects
Cases Sum of Squares df Mean Square F p ω²ₚ
Disgustingness 49.174 1 49.174 12.061 < .001 0.023
Disgustingness ✻ Gender 1.760 1 1.760 0.432 0.513 0.000
Residuals 346.552 85 4.077  
Frighteningness 138.075 1 138.075 32.122 < .001 0.064
Frighteningness ✻ Gender 5.523 1 5.523 1.285 0.260 6.263×10-4
Residuals 365.366 85 4.298  
Disgustingness ✻ Frighteningness 13.799 1 13.799 4.688 0.033 0.006
Disgustingness ✻ Frighteningness ✻ Gender 13.799 1 13.799 4.688 0.033 0.006
Residuals 250.177 85 2.943  
Note.  Type III Sum of Squares

The Within Subjects Effects table shows the results for the repeated measures factors. Disgustingness and Frighteningness both have a significant effect on Hostility (p < 0.001). There is also a significant two-way interaction between Disgustingness and Frighteningness, F(1,85) = 4.688, p = 0.033, and a significant three-way interaction between Disgustingness, Frighteningness, and Gender, F(1,85) = 4.688, p = 0.033. The effect size for the three-way interaction effect is quite small (i.e.,  = 0.006), while the effect sizes for the two significant main effects are also fairly small (i.e.,  = 0.023 and = 0.064).

Between Subjects Effects
Cases Sum of Squares df Mean Square F p ω²ₚ
Gender 18.346 1 18.346 0.996 0.321 0.000
Residuals 1565.879 85 18.422  
Note.  Type III Sum of Squares

The Between Subjects Effects table shows the results for the between-subjects factor Gender. There is no statistically significant main effect of Gender, F(1,85) = 0.996, p = 0.321. 

Descriptives

The plots below can be created in the Descriptive Plots menu by dragging Disgustingness to the horizontal axis, Frighteningness to Separate lines, and Gender to Separate plots. The plot shows the nature of the three-way interaction that was indicated by the hypothesis test: there seems to be no interaction effect between Frighteningness and Disgustingness for women (i.e., parallel lines), while there appears some interaction for men (i.e., non-parallel lines).

Descriptives plots

Frighteningness: Low
Frighteningness: High

Assumption Checks

Select Homogeneity tests from the Assumption Checks menu to test the assumption of equal variances between women and men on all 4 repeated measures (looking at the ratio of variances yields a similar conclusion). To assess normality, we can look at the Q-Q plot of residuals - most dots are aligned on the diagonal, with perhaps some non-normality in the tail-areas. Since the sample size is fairly big, this is no reason for concern though.

Test for Equality of Variances (Levene's)
  F df1 df2 p
Lo D, Lo F 0.453 1 85 0.503
Lo D, Hi F 0.598 1 85 0.441
Hi D, Lo F 0.527 1 85 0.470
Hi D, Hi F 0.042 1 85 0.839

Q-Q Plot

Post Hoc Tests

Lastly, there is a jungle of *Post Hoc Tests* that you can look at to get some more formal significance tests to investigate where the group differences lie. Below are three tables that make pairwise comparisons, each of the tables conditioning on a different variable. We will focus on the table that is conditional on Gender (first table below).


Here we get various comparisons, for men and women separately. What we can see, for instance, is that for both genders, there is a difference in hostility towards insects that score *low* on disgustingness and *low* on frighteningness, and insects that score *low* on disgustingness and *high* on frighteningness (2nd row; t = -5.089, *p* < .001 for women, t = -4.295, *p* < .001 for men).


Where the genders differ, however, is that women show a difference in hostility towards insects that score *high* on disgustingness and *low* on frighteningness, and insects that score *high* on disgustingness and *high* on frighteningness (5th row; t = -4.134, *p* < .001). This difference is not shown for men, who have similar levels of hostility towards both types of insects (t = -0.409, *p* = 1)

Post Hoc Comparisons - Gender ✻ Disgustingness ✻ Frighteningness - Conditional on Gender
Gender Mean Difference SE df t pholm
Woman Low, Low High, Low -0.647 0.375 85 -1.725 0.091
    Low, High -1.603 0.315 85 -5.089 < .001
    High, High -2.250 0.320 85 -7.041 < .001
  High, Low Low, High -0.957 0.432 85 -2.215 0.088
    High, High -1.603 0.388 85 -4.134 < .001
  Low, High High, High -0.647 0.319 85 -2.029 0.091
Man Low, Low High, Low -1.793 0.530 85 -3.382 0.004
    Low, High -1.914 0.446 85 -4.295 < .001
    High, High -2.017 0.452 85 -4.464 < .001
  High, Low Low, High -0.121 0.611 85 -0.198 1.000
    High, High -0.224 0.549 85 -0.409 1.000
  Low, High High, High -0.103 0.451 85 -0.230 1.000
Note.  P-value adjusted for comparing a family of 6 estimates.
Post Hoc Comparisons - Gender ✻ Disgustingness ✻ Frighteningness - Conditional on Disgustingness
Disgustingness Mean Difference SE df t pholm
Low Woman, Low Man, Low 0.793 0.607 85 1.307 0.390
    Woman, High -1.603 0.315 85 -5.089 < .001
    Man, High -1.121 0.590 85 -1.900 0.183
  Man, Low Woman, High -2.397 0.599 85 -4.004 < .001
    Man, High -1.914 0.446 85 -4.295 < .001
  Woman, High Man, High 0.483 0.581 85 0.831 0.409
High Woman, Low Man, Low -0.353 0.720 85 -0.491 1.000
    Woman, High -1.603 0.388 85 -4.134 < .001
    Man, High -0.578 0.618 85 -0.935 1.000
  Man, Low Woman, High -1.250 0.671 85 -1.863 0.329
    Man, High -0.224 0.549 85 -0.409 1.000
  Woman, High Man, High 1.026 0.560 85 1.832 0.329
Note.  P-value adjusted for comparing a family of 6 estimates.
Post Hoc Comparisons - Gender ✻ Disgustingness ✻ Frighteningness - Conditional on Frighteningness
Frighteningness Mean Difference SE df t pholm
Low Woman, Low Man, Low 0.793 0.607 85 1.307 0.443
    Woman, High -0.647 0.375 85 -1.725 0.353
    Man, High -1.000 0.684 85 -1.461 0.443
  Man, Low Woman, High -1.440 0.647 85 -2.226 0.143
    Man, High -1.793 0.530 85 -3.382 0.007
  Woman, High Man, High -0.353 0.720 85 -0.491 0.625
High Woman, Low Man, Low 0.483 0.581 85 0.831 1.000
    Woman, High -0.647 0.319 85 -2.029 0.274
    Man, High 0.379 0.567 85 0.669 1.000
  Man, Low Woman, High -1.129 0.574 85 -1.967 0.274
    Man, High -0.103 0.451 85 -0.230 1.000
  Woman, High Man, High 1.026 0.560 85 1.832 0.282
Note.  P-value adjusted for comparing a family of 6 estimates.