Structured Literature Review

Scientific method is warranted in the literature review just as much, if not more so, than in the studies themselves. It starts with being conscientious in the pursuit of a comprehensive list of studies that pertain to the topic, with an eye towards developing a balanced and comprehesive knowledge of the topic area and all viewpoints (even if they conflict with your own).
A structured approach to the literature review then helps to insure that important information is not overlooked. For example, findings that conflict may be telling you important things about conditions under which the treatment is most effective, implementation procedures necessary to produce the effect, etc.
Pillemer & Light (1980). have identified four general strategies for data synthesis:
Conducting a combined significance test from summary statistics;
Computing an average effect size (e.g. meta-analysis);
Investigating interactions between study attributes and outcomes, and;

Comparing similarly labeled treatments.
Conducting a Combined Significance Test
This can be useful in combining the results of two or more studies that (this is very important):

examined treatments that would not be expected to be dramatically different in outcome (but one might be preferred over the other for other reasons, such as cost)

tested the same directional hypothesis;

are clearly equivalent tests of the same directional hypothesis (i.e., outcome variation cannot be attributed to differences in context, participant characteristic, or some other factor that might logically be expected to interact with the treatment).

several studies tend to clearly favor one treatment over the other (or over the control), but the effect is not always significant.
This is a very simple procedure. All you need is the sample size and the value of the test statistic (e.g., t, z, F). If comparing z-scores, for example, the z's are added across studies and then divided by the sqare root of the combined sample sizes. Rosenthal (1978) discusses detailed procedures.

Computing an Average Effect Size
This is meta-analysis. It is covered in a separate section accesible via the research flowchart.
Meta-analysis is useful where the researcher is interested in developing an estimate of the degree of difference to be expected between the two treatments (or treatment versus control).
Meta-analysis, like combined significance, is appropriate only where the effects of the treatment are consistent. If some studies indicate a benefit due to the treatment and others indicate negative effects, the studies need to be examined very closely for contextual or participant factors that may interact with the treatment.

Investigating Interactions
The question the researcher attempts to answer here with the literature review is, "For whom, and under what conditions, is the treatment most effective?"
Means, sample sizes, and standard deviations are used to create mean squares and perform a two-way analysis of variance (ANOVA). If there is a significant interaction term (treatments-by-studies), this indicates that there may be moderating variables (e.g. context or participant characteristics) affecting the operation of the treatment effect. See Rosenthal (1978) for details.

Comparing Similarly Labeled Treatments
When you combine "similar" experiments, it is extremely important to insure that they really are similar in all important respects.
This technique is not simple. It involves "clustering" studies that report similar outcomes and then comparing clusters to try to determine why they are different (e.g., comparing clusters that report negative findings with clusters that report beneficial findings).

Choosing an Approach
The approach chosen should of course be consistent with the intent. Pillemer & Light (1980) go into more detail. The main things to remember:

Before you combine studies, make certain there is no compelling reason not to combine them (e.g., clearly different contexts or participants, dramatically different effect sizes, effects in different directions, etc.)

If you are interested in selecting among treatments that are very similar in effect but different in other respects (e.g. cost), combined significance testing is your probably choice.

If results tend to be consistent but varying in effect size, meta-analysis is probably your choice if you are interested in the likely "true" value of the effect size.

The technique of using experiments as blocking variables in an ANOVA can be useful if you suspect a treatment by experimental context interaction (i.e., the effect of the treatment depends on other things, such as context or participant characteristics).

Return to the research flowchart.