Answers to Study Questions

for

Chapter 10

(Don’t forget that the companion website also has multiple choice questions that you can take for practice. You will find them here: http://www.southalabama.edu/coe/bset/johnson/dr_johnson/2mcq.htm)

10.1. What is a quasi-experimental design, and when do you use such a design?

Quasi-experimental research designs are experimental designs that do not provide for full control of extraneous variables primarily because of the lack of random assignment to groups. They are stronger than the three “weak” designs discussed in the last chapter, but they are not as strong as the five “strong” designs that we discussed in the last chapter.

· You could say that they are kind of “in between” designs; they are not great, but they are not too bad either. Because they are classified as a type of experimental research, the independent variable must manipulated (although real world events that are highly similar to experimenter manipulation also may be appropriate for quasi-experimental research).

• You can view quasi-experiments as falling in the center of a continuum with weak experimental designs on the far left side and strong experimental designs on the far right side. (In other words, quasi designs are not the worst and they are not the best. They are in-between or moderately strong designs.)

/------------------------------------/------------------------------------/

Weak Quasi Strong

Designs Designs Designs

• Quasi-experimental research designs are used when a) you are interested in studying cause and effect, b) you can manipulate the independent variable, c) you are not able to use a stronger experimental design because of practical or other constraints.

10.2. What requirements must be met to reach a valid causal inference when using

a quasi-experimental design?

Basically, you must meet the same three necessary conditions that we discussed in the last chapter. Here are these extremely important conditions again presented in a Table from a later chapter in your book. Again, whenever you are trying to establish a cause and effect relationship as being present you must, at a minimum, do these three things:

You must memorize these three conditions and use them whenever you are thinking about cause and effect in research.

· Quasi-experimental research is strong on conditions one and two.

· Therefore, condition three is where you will have to do the most thinking and planning when you design a quasi-experimental research study.

10.3. What is a nonequivalent comparison-group design and what are the essential features of this design?

O₁ X₁ O₂

-------------------------

O₁ X₂ O₂

where

O₁ and O₁ are the pretests

X₁ and X₂ are the levels of the independent variable

and O₂ and O₂ are the posttests.

Here are the essential features of this design:

· Manipulation of the independent variable

· Pretest for all of the comparison groups

· Posttest for all of the comparison groups

· No random assignment to the comparison groups (which as you can imagine is going to cause some problems with this design as compared to the strong “randomized” experimental designs.

10.4. How are rival explanations addressed when using the nonequivalent comparison-group design?

First, when designing a study that will be based on this design, you must try to identify groups that are as similar as possible on any factors that may affect the dependent variable. Next, ask yourself this question: “Looking at these groups, do I have any reason to expect them to be different in any way?” If yes, ask what are those characteristics? and how can you control for them? The logic here is that you don’t want the groups to differ on any extraneous variables that may affect the dependent variable. You want the groups to be as similar as possible on all factors except the independent variable. That is, you want the only systematic difference between the groups to be the systematic variation of the independent variable.

The biggest threat to the internal validity of these designs is selection (i.e., the groups might be composed of different kinds of people with different characteristics such as group differences in age, gender, IQ, reading ability, etc.). Other threats are mortality and the selection interactions such as the selection-maturation effect, and the selection-history effect.

You need to try to control for all extraneous variables on which the groups may differ using control techniques such as statistical control (e.g., analysis of covariance) and matching.

10.5. What types of biases can exist when using the nonequivalent comparison-group

design?

Here is a list of the most common types of biases that can exist in the nonequivalent-comparison group design. They are all potentially important and must be addressed by a researcher. In my opinion (Burke Johnson) the omnipresent and most serious problem is selection bias (what we called earlier differential selection); this is always a problem in this design because you do not have random assignment to the groups, and therefore the groups may differ on some extraneous variables and not just differ on the levels of the independent variable that they get. In other words, you must always be concerned about differences in the makeup of the comparison groups.

10.6. What is the best way of determining whether a threat is plausible when using

the nonequivalent comparison-group design?

1. First, you should examine the way in which the participants were assigned to the groups and try to determine what variables the groups may differ on.

· If you identify any extraneous variables that the groups differ on and which are probably also related to the dependent variable then you should consider these to be plausibly alternative explanations, unless they are addressed and controlled by the researcher.

2. Second, you should look for pretest differences because big pretest differences will tend to lead to big posttest differences regardless of any posttest differences caused by the independent variable.

· If the researcher did not use any technique to control for pretest differences, then this should be considered to be a plausible alternative explanation (i.e., a threat).

3. Third, all of the other potential biases shown above in Table 10.1 must be considered.

· If any seems likely to be present and nothing was done to minimize its impact, then you should consider it to be a plausible threat to the study.

The bottom line is this: Was the threat identified by the researcher?, and did the researcher use any of the control techniques (e.g., statistical control such as analysis of covariance and matching) to minimize the problem?

· If the answer is NO, then any threat identified by you as being plausible is a problem and the researcher will need to explain why he or she did not attempt to address the problem during the planning and conduct of the study and what this means for the study results.

10.7. What are the essential design characteristics of an interrupted time-series design?

The interrupted time-series design is shown in the following figure from your textbook:

As you can see above, the three major design components are

· multiple pretests (it does not have to be exactly five as shown in the picture)

· a treatment (i.e., an intervention to be studied)

· multiple posttests (it does not have to be exactly five as shown in the picture)

Note that the type of treatment can be one that occurs specifically between the time points 5 and 6, but the type of intervention can also be one that is “turned on” after time point 5 and continues throughout the posttesting period (i.e., throughout points 6 to 10).

10.8. How is a treatment effect demonstrated when using an interrupted time-series

design?

It is demonstrated by comparing the pretest pattern (its height, slope, and shape) with the posttest pattern. Note that the pretest pattern shows the baseline (i.e., the pattern before any intervention).

Here is an example of a clear pattern of results, suggesting that the treatment has an impact on the dependent variable:

10.9. How are potential confounding variables ruled out when using the interrupted

time-series design?

The key strategy of this design is to have enough pretests and posttests to determine the pattern of results on the dependent variable (i.e., on the variable that you are trying to influence). By including multiple pretests and posttests you are able to avoid many of the problems present in the weak design called the one-group pretest-posttest design that was discussed in the last chapter. The one-group pretest-posttest design has only one pretest and one posttest.

If you look at the following figure, you will see that the one-group pretest-posttest design would give misleading results in cases A, B, and C, but the interrupted time-series design provides the data in all of the cases that enables you to make an accurate determination of the impact (or lack of impact) of the treatment or intervention.

· Visual inspection is sometimes used to make a determination of impact; however, if there is much variation in the data, statistical analysis is usually needed in order to provide high statistical conclusion validity.

· Note that the primary threat to the validity in an interrupted time-series design is the history effect; that is, if some additional factor occurs after the baseline period along with the treatment then this represents a rival explanation (for example, if you were studying the impact of a change in the speed limit on traffic fatalities you would not want to select localities where a new drunk driving law was implemented at the same time as the speed limit change).

10.10. What are the essential characteristics of the A-B-A design?

Here is a picture of the A-B-A design:

The A-B-A design is a single-case experimental design in which the response to the experimental treatment condition (B) is compared to baseline responses (A) taken before and after administering the treatment condition.

· The treatment effect in this design is determined by comparing the pattern of baseline behavior with the pattern of treatment behavior.

· You would look for a discontinuity in the baseline and treatment patterns, such as differences in the slope and/or level of responses on the dependent variable.

· For example, if the treatment is supposed to cause an increase a certain behavior then an impact would be shown by a low (A), high (B), low (A) set of responses.

· If the treatment is supposed to cause a decrease a certain behavior then an impact would be shown by a high (A), low (B), high (A) set of responses.

10.11. How does the A-B-A design rule out rival hypotheses and demonstrate the effect of

an experimental treatment condition?

If you used an A-B design rather than an A-B-A design, history would be a threat (i.e., anything that co-occurred with the treatment). However, if you use the A-B-A design then history is likely to be ruled out through what is called reversal. Reversal occurs when the behavior to reverts back to baseline when the treatment is removed. It is unlikely that some extraneous history event or variable would start exactly the same time as the treatment, occur during the treatment, and then precisely disappear when the treatment is removed. That’s why reversal is so useful.

· Short treatment periods help to facilitate reversal in some situations.

10.12. What are the primary problems that can exist when using the A-B-A design, and

how can they be solved?

The primary limitation with the A-B-A design is that some behaviors will not revert back to the original baseline level when the treatment is removed, making rival hypotheses such as history more plausible.

· To solve this problem you need to use a different design, in particular, the multiple-baseline design.

A second limitation is that the experiment ends with the baseline condition, and this means that the experiment does not end by providing the benefits of the treatment.

· This limitation is overcome by using an A-B-A-B design rather than an A-B-A design.

10.13. How does the multiple-baseline design demonstrate a treatment effect?

· First, note that in the multiple-baseline design, the treatment condition is successively administered to different participants (or to the same participant in several settings) after baseline behaviors have been recorded for different periods of time. In other words, the treatment has a delayed or staggered onset for the different participants.

· Notice that the effectiveness of this design does not hinge on a reversal of behavior (as was the case in the A-B-A and A-B-A-B designs).

· The “fingerprint” or pattern of results that suggests an effect would be a change in behavior (in the predicted direction) after each onset of the treatment.

· For example, the following pattern of results supported the researchers’ theory (i.e., the reading rates would increase when they implemented their assisted reading treatment for each of three participants).

· Notice how the onset of the treatment was staggered (i.e., occurs at different times) for the different participants in the above research study and notice the pattern of results.

10.14. What is the primary problem that can be encountered in using the

multiple-baseline design?

A limitation is that some behaviors, people, or settings may be interdependent; that is, when you change one person, behavior, or setting another person, behavior, or setting being measured in the study may also be affected. When this happens you don’t get the pattern of responses needed to draw a firm conclusion about the effectiveness of the treatment (because in this design changes in the measure should clearly occur with each onset of the treatment condition).

10.15. When would you use the changing-criterion design?

The changing-criterion design is used when the researcher wants to demonstrate that the participant’s behavior can be altered by changing the criterion for success during successive treatment periods. For example, a researcher might want to “shape” a participant’s behavior or the researcher might want to increase the accuracy, frequency, or amount of a behavior over time.

10.16. What are the essential characteristics of the changing-criterion design?

The characteristics are shown in the following figure from your textbook:

This type of design includes a baseline phase and several treatment phases where the criterion of required performance is successively increased or decreased.

· (Take a look at Figure 10.16 to see a pattern of results that supported the researcher’s hypothesis.)

Here is one more study question that was not included in your book, but I want to add...

10.17 Identify and discuss the four methodological issues that must be considered when using a single-case research design.

The four issues are baseline (e.g., When has a stable baseline been reached? How much variability or fluctuation is present? Is there a trend present? What is the direction of the trend?), changing one variable at a time (Make sure you only change one variable at a time so that no confounding of causes will be present), length of phases (How long should a phase be? Generally, all phases should be the same length), and assessment of treatment effect (How do you decide if the treatment caused a change in the dependent variable? Visual inspection and statistical analysis are used for this).