Chapter 9

Answers to Study Questions

9.1. What is an experiment, and what are the significant components of this definition?

The experiment is a situation in which a researcher attempts to objectively observe phenomena which are made to occur in a strictly controlled situation where one or more variables are varied and the others are held constant.

· The definition says that we should attempt to make impartial and unbiased observations in the experimental situation. In short, objectivity is the ideal to which experimenters strive even though perfect objectivity is impossible to achieve.

· In experiments phenomena are made to occur. The phenomena are observable events; they are the conditions presented to the participants. Specifically, these phenomena or conditions are the levels of the independent variable that are made to occur (e.g., one group is given a pill and another group is given a placebo). This is the idea of active manipulation. The idea is that an experimental researcher does something and then observes the outcome. (Manipulation is the key defining characteristic of an experiment, as you learned in chapter 2.)

· The observations in the laboratory experiment are made under conditions set up and controlled by the researcher; if the experiment has multiple groups then the researcher attempts standardize the conditions for all groups with the only difference being that the different groups get different levels of the independent variable. The key idea is that the researcher tries to set up a situation where the ONLY systematic difference between the groups to be that they got different levels of the independent variable.

· The researcher attempts to hold all variables other than the independent variable constant. This is best done by first, randomly assigning participants to the groups (which will “equate” the groups on all known and unknown variables at the beginning of the study), and second, by standardizing the conditions as much as possible so that the only difference that occurs during the experiment is the administration of the levels of the independent variable.

Here is some additional information: Experiments, like any other approach that hopes to deal with causality, must meet the three necessary conditions for establishing cause and effect: 1) establish a relationship, 2) check for proper time order, and 3) make sure there are no plausible alternative explanations. The “strong” experimental designs discussed in this chapter are the best arrangement we have for isolating causal effects and providing evidence of cause and effect. If you have a comparison group and random assignment, for example, then you will be able to see if there is a relationship between the independent and dependent variables (just check to see if the group means become different after the treatment), you will be able to establish proper time order (because you can observe the dependent variable after the independent variable is administered), and you have ruled out any alternative explanation that suggests that the difference between the groups at the end of the experiment is due to differences in the initial composition of the groups (because when you randomly assign participants to groups, the groups will, on average, be equivalent). Remember that random assignment (if present) equates the groups (in a probabilistic way) on all extraneous variables at the beginning of the experiment. That’s the “secret ingredient” of the strongest experimental designs. It’s the secret condition that you should always try to use if possible.

Hers is some more terminology that is important: an independent variable is a variable that is presumed to cause a change in another variable, and in experimental research, the independent variable is the variable that is manipulated by the researcher. The dependent variable is a variable that is presumed to be influenced by one or more independent variables. The basic aim of an experiment is to show that changes in the manipulated IV cause changes to occur in the DV.

9.2. What are the different ways a researcher can use to manipulate an independent variable?

Figure 9.1 shows three ways the IV can be manipulated. First, the IV can be manipulated by presenting a condition or treatment to one group and withholding the condition or treatment from another group (the presence or absence technique). For example, the researcher may give a new drug to one group and a placebo (a pill with no active ingredient) to the control group. Second, the IV can be manipulated by varying the amount of a condition or variable (the amount technique). For example, the researcher may provide three levels of instruction to the participants in three groups (none, one hour, and five hours). Third, the IV can be manipulated by varying the type of the condition or treatment administered (the type technique). For example, the researcher may provide client-centered counseling to one group of depressed patients and provide rational-emotive therapy to the other group of depressed patients.

9.3. What is meant by the term experimental control, and how is experimental control related to differential influence within the experiment?

Experimental control refers to the researcher’s attempt to eliminate any differential influence of extraneous variables. In multi-group designs, the key is to eliminate the problem of differential influence of any and all extraneous variables; this is done by making the groups similar on any important extraneous variables at the start of the experiment and during the experiment (i.e., it is done by achieving experimental control). In other words, you want the extraneous variables (the bad variables that can confound or confuse your conclusions) to have a constant effect on the DV (i.e., you want to eliminate the problem of differential influence).

For example, if IQ is related to the DV, you want make sure that the different groups are composed of people with similar IQ levels so that the groups will not differ on the extraneous IQ variable. Also, during the conduct of the experiment it is important to treat all groups the same (except for administration of the IV). (In the next chapter we will talk about how to obtain some control for extraneous variables in single-group and single-case experimental designs.) In multi-group designs (designs with two or more comparison groups) the goal is to make the groups the same on all extraneous variables which might have an influence on the DV through the operation of differential influence. Then we systematically vary the levels of the IV. Finally, if we did this, we will be able to conclude that the group differences that emerge are due to the IV rather than due to some extraneous variable. In short, we will be able to conclude that the IV affected the DV, and that what we observed was not due to some other variable. We will have established the causal link between the IV and DV (which is the goal of experimental research).

In summary, differential influence is bad and we want to eliminate it. We eliminate it through experimental control, which is good.

9.4. What is random assignment, and what is the difference between random assignment and random selection?

Random assignment is the strongest of all of the experimental control techniques; by randomly forming groups, the groups will be probabilistically equated on all known and unknown variables at the start of the experiment. As you can see, this is very powerful. Random selection is very different from random assignment. The purpose of random selection is to generate a sample that represents a larger population. The purpose of random assignment is to take a sample (usually a convenience sample) and randomly divide it into two or more groups that represent each other. Using a mirror metaphor, in random sampling, we want to sample to mirror the population, but in random assignment we simply want the different groups to mirror one another. By the way, in experimental research, random assignment is much more important than random selection; that’s because the purpose of an experiment to establish cause and effect relationships.

9.5. How does random assignment accomplish the goal of controlling for the influenced of extraneous or confounding variables?

Random assignment starts with a group of research participants. Then using the process of random assignment, these participants are randomly assigned to two or more groups. Random assignment means that the researcher is taken out of the loop of making decisions about who goes into the different groups. Instead, the mathematical theory of probability is used to conduct random assignment.

Because random assignment is so important, here is a little more about it...Random assignment “equates the groups” on all known and unknown extraneous variables at the start of the experiment. This makes it very plausible that any significant observed difference between the groups on the DV after the administration of the IV can be attributed to the effect of the IV. Even when you use another control technique (e.g., matching or analysis of covariance) the experimental research design is dramatically improved if random assignment is also used. Random assignment is what I (Johnson) call the “secret ingredient” of a strong experimental design. If you can include that ingredient in your research design, then include it!

Here is one way you can carry out random assignment (it was included in the first edition of this book):

Another way is to simply go to this website and use the computer assignment generator: http://www.graphpad.com/quickcalcs/randomize1.cfm Check it out to see how it works.

9.6. How would you implement the control technique of matching, and how does this technique control for the influence of confounding variables?

In matching you have to decide on the specific variable or variables that you want to use to equate the groups on. For example, let’s say that you decide to equate your two groups (treatment and control group) on IQ. What you would do is to rank order all of the participants on IQ. Then select the first two (i.e., the two people with the two highest IQs) and put one in the experimental treatment group and the other in the control group (The best way to do this is to use random assignment to make these assignments. If you do this then you have actually merged two control techniques: matching and random assignment). Then take the next two highest IQ participants and assign one to the experimental group and one to the control group. Then just continue this process until you assign one of the lowest IQ participants to one group and the other lowest IQ participant to the other group. Once you have completed this, your two groups will be matched on IQ! If you use matching without random assignment, you run into the problem that although you know that your groups are matched on IQ you have not matched them on other potentially important variables.

9.7. How would you use the control technique of holding the extraneous variable constant?

In this technique, you decide on the variable you want to hold constant such as gender. Then you only use one gender in the study (e.g., you decide to use only females or only males). Since all the people in the different groups are of the same gender, differential influence is impossible. Note that a big problem with this control technique is that it is hard to generalize after you have systematically excluded an important group of people. You have improved your status on internal validity, but your have hurt your status on external validity. In some studies, e.g., studies on breast cancer or prostrate cancer, this is not a problem because generalization is only appropriate for one group anyway.

9.8. When would you want to build the extraneous variable into the research design?

When you want to control for it and also want to study how it is related to the outcome variable (DV). By including the extraneous variable, you can check for its main effect as well as its interaction effect (see pages 286-287 for explanation of these two terms). Basically, when you include the extraneous variable, you learn more about it and you are able to control for the problems that it might have otherwise caused.

9.9. What is counterbalancing, and when would you use it?

Counterbalancing is only used for a certain type of design: repeated measures designs or mixed designs (i.e., designs that have a repeated measures variable). In the repeated measures design, all participants receive all levels of the independent variable (which is different from all of the other designs we have been talking about where the groups are composed of different people). When all participants receive all treatments you should be concerned that the sequencing of the treatments may also have an impact. The two specific sequencing effects (biasing effects that can occur because each participant must participate in more than one treatment condition) that may occur are order effects (participants perform differently because of the order in which they receive the treatment) and carryover effects (participants’ performance in a later treatment is different because of a treatment that occurred prior to it). To help minimize the impact of sequencing effects, your would use the control technique called counterbalancing. What this means is the you administer the experimental treatment conditions to a the people but you do it in different orders for different sets of people. For example if the IV were teaching method and it included two teaching methods for comparison you could counterbalance by giving half of your participants this order Method1 followed by Method 2 and by giving the other half of your participants this order Method 2 followed by Method 1. In education, counterbalancing typically will need to be used whenever you use a repeated measures design. Note that when counterbalancing is used for the repeated measures design, it becomes a strong design (in Table 9.2 you can see that the question marks turn into plus signs when counterbalancing is used).

9.10. What is the difference between a carryover effect and an order effect?

I answered that in the previous question.

9.11. What is analysis of covariance, and when would you use it?

Analysis of covariance is a statistical control method that is used with multi-group designs. In particular, it is used to statistically equate groups that differ on a pretest or on any other extraneous variable that you are worried about and you can measure.

9.12. What is a research design, and what are the elements that go into developing a research design?

A research design is the outline, plan, or strategy that is used to answer a research question. You can see three weak experimental designs in Table 9.1 and five strong experimental designs in Table 9.2. The primary elements going into these designs are decisions about whether to use pretests, control groups, how many pretests and posttests to use, whether to randomly assign or use some other control technique, how you are going to collect your data (i.e., how you are going to measure your variables). The designs shown in Tables 9.1 and 9.2 (and the quasi and single case designs discussed in the next chapter) all have slightly different combinations of groups and testing and use of control techniques. In fact, if you are creative, you can construct your own design from these components, but first you need to think about the designs we discuss.

For your convenience, here are Tables 9.1 and 9.2.

9.13. When would the one-group posttest-only design be used, and what are the problems encountered in using this design?

This design is so poor that I would recommend never using it. The biggest problem is that you have little evidence that what is observed at the posttest is due to the treatment because you have no pretest to use as a baseline measure (i.e., you don’t know where they started out). This is the weakest of all experimental designs. The only possible situation I can think of where this design might be of some use would be where the posttest measures something that you are pretty sure they all knew nothing about previously so that you can assume that if a pretest had been given then they all would have scored very low. Occasionally in some training situations this assumption might be made. Still, you can very much improve this design by adding a pretest (i.e., transforming it into the one-group pretest-posttest design). The key threats to this design (in addition to the problems I just mentioned) are history and maturation.

9.14. When would you use the one-group posttest-only design, and what are the potential rival hypotheses that can operate in this design?

You can use this design when you don’t need real strong evidence about cause and effect and when you want to measure whether people change over time due to a treatment. For example, if you want to train the workforce in your organization to understand their retirement system, you could pretest them, give the training, and then posttest them and look for an increase in knowledge about the retirement system. The key threats to internal validity for this design are history, maturation, testing, instrumentation, and regression artifacts.

9.15. When would you use the posttest-only design with nonequivalent groups, and what are the potential rival hypotheses that can operate in this design?

This design would be used when you are not able to do pretesting but you can get a comparison or control group. A serious limitation with this design is that because you do not have random assignment to the groups you cannot assume that the groups are similar on any variables, and therefore, it is highly risky to conclude that any difference observed on the posttest is the result of the treatments. The key threats to the internal validity of this design (in addition to the problem I just mentioned) are differential selection, differential attrition, and additive and interactive effects.

9.16. What makes a design a strong experimental design?

In most cases, that key is RANDOM ASSIGNMENT to the groups, which equates the groups on all known and unknown extraneous variables. When you are confident that the groups are composed of similar kinds of people, many threats to internal validity disappear.

We have listed five designs as being “strong” designs: pretest-posttest control-group design, posttest-only control-group design, factorial design, repeated-measures design, and the factorial design based on a mixed model. The key ingredient of all of these designs except the repeated-measures design is that random assignment is present (which equates the groups on all known and unknown extraneous variables). The repeated measures design and the factorial design based on a mixed model are considered strong only if counterbalancing is used to eliminate or minimize order and carryover effects.

You can see the dramatic change in the threats for the strong designs by comparing Table 9.1 and 9.2 shown above. Note How there are no minus signs for the standard threats to internal validity for the strong designs shown in Table 9.2!

9.17. What is the difference between an experimental and a control group?

An experimental group receives the experimental treatment. A control group does not receive the experimental treatment condition. The standard in experimental research is to compare an experimental group with a control group. However, sometimes two experimental or comparison groups are compared with each other. You can also have a design where you compare different treatments and have a control group.

9.18. What functions are served by including a control group into a research design?

A control group gives a point of comparison. For example, if one group receives a pill you also need to know what would have happened in the absence of receiving a pill, and the group receiving a placebo (a pill with no active ingredient) serves this purpose. It is essential, however, that the control group be similar to the experimental group on all important characteristics (this is where random assignment helps out things). As an aside, some researchers call the point of comparison the counterfactual (which is the pretest level in a one-group design with a pretest measure and the control or comparison group performance in a multi-group design).

· It is perhaps easiest to see the effect of including a control group by comparing the one-group pretest-posttest design and the posttest-only design with nonequivalent groups design shown above in Table 9.1.

· Notice that these threats disappeared when the control group was added: history, maturation, testing, instrumentation, and regression artifacts.

· However, notice that new potential threats occurred when the control group was added; now (because you have two or more groups) you have to worry about differential selection, differential attrition, and additive and interactive effects.

9.19. What potentially confounding extraneous variables are controlled in the pretest-posttest control-group design?

All of the standard threats to internal validity are controlled in this design. It is because of one important factor included as part of this design: RANDOM ASSIGNMENT!

9.20. What potentially confounding extraneous variables are controlled in the posttest-only control-group design, and how does the design control for them?

This design is similar to the pretest-posttest control-group design. All of the standard internal validity threats are ruled out because this design has RANDOM ASSIGNMENT. As you can see, you don’t really have to have a pretest if you have random assignment. Random assignment really is a powerful technique of control.

9.21. What is a factorial design, and what is the advantage of this design over the two-group posttest-only design (e.g., the posttest-only control-group design with two groups)?

The posttest-only control group design has only one independent variable. The factorial design has at least two independent variables and random assignment to the groups (or cells), which enables you to determine whether the single independent variables have effects (called main effects) and, importantly, whether there is an interaction effects (i.e., do the two independent variables interact in some way that would make knowledge only of main effects misleading?) All you can determine in the posttest-only control group design is whether there is a main effect, and sometimes this limitation can cause a problem if a moderator variable has been excluded. In short, the factorial design is a superior design.

9.22. What is a main effect?

A main effect is the effect of one independent variable.

9.23. What is an interaction effect, and what is the difference between an ordinal and a disordinal interaction?

An interaction effect is what occurs when the effect of one independent variable (on the DV) depends on the level of another independent variable. For example, perhaps one teaching method works better than another teaching method depending upon the students’ learning styles. Note that in Chapter 2, in Table 2.2 we use the term moderator variable to refer to an independent variable that determines how another IV and the DV are related. In other words, when you have an interaction effect, you also have a moderator variable present. Interactions are helpful because them help use to better understand form whom or under what conditions various treatments will work.

Here is an example where there is no interaction (note that the lines are parallel which means that the effect of instruction type does NOT depend on anxiety):

Note: There is an interaction effect present when the effect of one independent variable on the dependent variable changes or varies at the different levels of another independent variable. If you have to say that the your conclusion about the relationship between an IV and a DV “depends” on the level of another IV, then you have an interaction effect. For example, perhaps the relationship between teaching technique and student learning depends on the type of student being taught. (By the way, technically speaking, I am talking about what is called a two-way interaction effect here.)

· Now I will show two cases where there is an interaction effect.

First, here is an example of a disordinal interaction (not only are the lines nonparallel, they also cross): Notice that the type of instruction to be recommended depends on the moderator variable of anxiety.

Second, here is an example of an ordinal interaction (the lines are nonparallel but do not cross in the range examined in the research study): Notice that the type of instruction to be recommended depends on the moderator variable of anxiety.

9.24. What is the difference between a factorial and a repeated measures-design?

A factorial design has two or more independent variables of the type that different people are placed in the different levels of the independent variables. In contrast, in the repeated-measures design, all participants participate in all levels of the independent variable.

9.25. What are the advantages and disadvantages of factorial and repeated-measures designs?

The advantages of the factorial design are that you can examine main effects and interaction effects. The main advantage of the repeated measures design is that you can get by with fewer participants because you use your participants in all of the groups rather than having to have different participants in the different groups.

9.26. What is a factorial design based on a mixed model, and when would it be used?

This designs is a mix between the factorial design and the repeated measures design. It has one regular independent variable (where different people are randomly assigned to the different levels), and it has one repeated measures variable (i.e., a variable in which all participants receive all the treatments).

Here is a depiction of the factorial design based on a mixed model.