Meta-analysis is a structured literature review technique with the goal of combining similar studies to determine the average effect size for a particular treatment under comparable circumstances with comparable participants.

Fitz-Gibbon and Morris (1987) provide a worksheet for doing a meta-analysis. Briefly:

  1. Identify the effect size that is to be investigated.

    Note: the SD of Y is the pooled standard deviation, the overall SD of experimental and control groups combined.

  2. Develop coding sheets.

    Read each study, being very careful to take notes anytime you see something that you think might have an impact on the result of the experimental treatment. Create a table with studies down the left side and factors across the top. For each study, place a check mark in the factor column if that factor was present in the study. Select only studies that you consider to be comparable on everything that might impact the outcome.

  3. Extract effect sizes.

    If all of the studies report group means and standard deviations, you can use the basic formula above. Otherwise, You can extract it from either the t value:

    or the correlation value:

  4. Compute standard errors.

  5. Graph and ruminate.

    Notice that studies 1, 2, and 4 are similar in effect size, and their effect sizes plus or minus one standard error overlap. Study 3 effect size, however, is larger than the other 3, and it's effect size plus or minus 1 standard error does not overlap with the other 3. This may or may not be meaningful, but it does indicate that you might want to take a very hard look at Study 3 to see if you can find anything about it that makes it different from the other three in terms of moderating factors on the impact of the treatment.

    
    
    
    
    
    
    
    
    
    
  6. Compute a mean effect size.

    This can be a simple unweighted mean, or greater weight can be given to studies with larger sample sizes. The unweighted mean is simply the sum of the effect sizes divided by the number, 1.02 in this case.

  7. Test for homogeneity of the effect sizes.

    The homogeneity statistic is distributed as a chi-square variable with degrees of freedom equal to the number of studies minus 1. If the computed value is larger than the tabular value for chi-square, then this suggests heterogeneity (effect sizes not homogeneous). In the example, the chi-square value for 3 degrees of freedom at the p<.05 level of significance is .781. In other words, our earlier feeling that something different was going on in Study 3 was probably right.

    
    
    
    
    
    
    
    
    
    
    
  8. Relate the effect size findings to the data on the coding sheets.

    In this example, you would want to go back to your coding table and take another look at Study 3. Did you overlook something? Do you need to add something to the coding sheet? This outlier study could merely have been done better than the other three, or there may have been some condition in the context or participants that led to this larger effect size.

  9. Interpret the size of the effect by reference to other information.

    You're on your own from this point. It is up to your professional knowledge about the treatment of interest and anything you know or suspect may be related to it. If you suspect you know, for example, why the effect size was so much larger in Study 3, you may decide to perform an experiment to test this suspicion (hypothesis).


Return to the research flowchart.