Chapter 11
Nonexperimental Quantitative Research

(Reminder: Don’t forget to utilize the concept maps and study questions as you study this and the other chapters.)

 

Nonexperimental research is needed because there are many independent variables that we cannot manipulate for one reason or the other (e.g., for ethical reasons, for practical reasons, and for literal reasons such as it is impossible to manipulate some variables).  Here’s an example of an experiment where you could not manipulate the independent variable (smoking) for ethical and practical reasons: Randomly assign 500 newborns to experimental and control groups (250 in each group), where the experimental group newborns must smoke cigarettes and the controls do not smoke.

 

Nonexperimental research is research that lacks manipulation of the independent variable by the researcher; the researcher studies what naturally occurs or has already occurred; and the researcher studies how variables are related.

 

Steps in Nonexperimental Research

The pretty much the same as they were in experimental research; however, there are some new considerations to think about if you want to be able to make any cause and effect claims at all (i.e., that an IV--->DV).

 

  1. Determine the research problem and hypotheses to be tested. Note: it is important to have or develop a theory to test in nonexperimental research if you are interested in making any claims of cause and effect. This can include identifying mediating and moderating variables (see Table 2.2 on page 36 for definitions of these two terms).

 

  1. Select the variables to be used in the study. Note: in nonexperimental research you will need to include some control variables (i.e., variables in addition to your IV and DV that measure key extraneous variables). This will help you to help rule out some alternative explanations.
     
  2. Collect the data. Note: longitudinal data (i.e., collection of data at more than one time point) is helpful in nonexperimental research to establish the time ordering of your IV and DV if you are interested in cause and effect.
     
  3. Analyze the data. Note: statistical control techniques will be needed because of the problem of alternative explanations in nonexperimental research.

 

  1. Interpret the results. Note: conclusions of cause and effect will be much weaker in nonexperimental research as compared to strong experimental and quasi-experimental research because the researcher cannot manipulate the independent variable in nonexperimental research.

 

When examining or conducting nonexperimental research, it is important to watch out for the post hoc fallacy (i.e., arguing, after the fact, that A must have caused B simply because you have observed in the past that A preceded B).

 

Independent Variables in Nonexperimental Research

This includes variables that cannot be manipulated, should not be manipulated, or were not manipulated.

 

 

Simple Cases of Causal-Comparative and Correlational Research

Although the terms causal-comparative research and correlational research are dated, it is still useful to think about the simple cases of these (i.e., studies with only two variables).  There are four major points in this section:

 

  1. In the simple case of causal-comparative research you have one categorical IV (e.g., gender) and one quantitative DV (e.g., performance on a math test).

·        The researcher checks to see if the observed difference between the groups is statistically significant (i.e., not just due to chance) using a "t-test" or an "ANOVA" (these are statistical tests discussed in a later chapter; they tell you if the difference between the means is statistically significant; they are discussed in chapter 16).

 

  1. In the simple case of correlational research you have one quantitative IV (e.g., level of motivation) and one quantitative DV (performance on math test).

·        The researcher checks to see if the observed correlation is statistically significant (i.e., not due to chance) using the "t-test for correlation coefficients" (it tells you if the relationship is statistically significant; it is discussed in chapter 16).

·        Remember that the commonly used correlation coefficient (i.e., the Pearson correlation) only detects linear relationships.
 

3.   It is essential that you remember this point: Both of the simple cases of nonexperimental research are seriously flawed if you are interested in concluding that an observed relationship is a causal relationship.

·        That's because "observing a relationship between two variables is not sufficient grounds for concluding that the relationship is a causal relationship."  (Remember this important point!)

 

4.   You can improve on the simple cases by controlling for extraneous variables and designing longitudinal studies (discussed below).

·        And once you move on to these improved nonexperimental designs, you should drop the “correlational” and “causal-comparative” terminology and, instead, talk about the design in terms of the research objective and the time dimension (which is discussed below, and summarized in Table 11.3)

 

The Three Necessary Conditions for
Cause-and-Effect Relationships

It is essential that your remember that researchers must establish three conditions if they are to make a defensible conclusion that changes in variable A cause changes in variable B. Here are the conditions (which have been stated in previous chapters) in a summary table:

 

 

 

Applying the Three Necessary Conditions

 for Causation  in Nonexperimental Research

Nonexperimental research is much weaker than strong and quasi experimental research for making justified judgments about cause and effect.

·        It is, however, quite easy to establish condition 1 in nonexperimental research—just see if the variables are related  For example, Are the variables correlated? or Is there a difference between the means?.

·        It is much more difficult to establish conditions 2 and 3 (especially 3).

·        When attempting to establish condition 2, researchers use logic and theory (e.g., we know that biological sex occurs before achievement on a math test) and design approaches that are covered later in this chapter (e.g., longitudinal research is a strong design for establishing proper time order).

·        Condition 3 is a serious problem in nonexperimental research because it is always possible that an observed relationship is "spurious" (i.e., due to some confounding extraneous variable or "third variable").

·        When attempting to establish condition 3, researchers use logic and theory (e.g., make a list of extraneous variables that you want to measure in your research study), control techniques (such as statistical control and matching), and design approaches (such as using a longitudinal design rather than a cross-sectional design).

·        The rest of the chapter will be explaining these points.

·        To get things started, you need to understand the idea of controlling for a variable. Here is an example: first, Did you know that there is a correlation between the number of fire trucks responding to a fire and the amount of fire damage? Obviously this is not a causal relationship (i.e., it is a spurious relationship). In Figure 11.2 below, you can see that after we control for the size of fire, the original positive correlation between the number of fire trucks responding and the amount of fire damage becomes a zero correlation (i.e., no relationship).
 

 


 

 

 

 

 

 

 

Techniques of Control in Nonexperimental Research

We discuss three ways to control for extraneous variables in nonexperimental research.

  1. Matching.


You could do this by finding people for each of the cells of the following table: 

 

 Low
Motivation 

Medium
Motivation

 High
Motivation

Low GPA

15 people 

15 people 

15 people

Medium GPA

15 people

15 people

15 people

High GPA

15 people

15 people

15 people

 

 

  1. Holding the extraneous variable constant.

 

  1. Statistical control (it's based on the following logic: examine the relationship between the IV and the DV at each level of the control/extraneous variable; actually, the computer will do it for you, but that’s what it does).

 

 

Now I am going to talk about the two key dimensions that should be used in constructing a nonexperimental research design: the time dimension and the research objective dimension. (Note that these dimensions eliminate the need for the terms correlational and causal-comparative in nonexperimental research.)

 

 

The Time Dimension in Research

Nonexperimental research can be classified according to the time dimension. In particular, Figure 11.3 shows and summarizes the three key ways that nonexperimental research data can vary along the time dimension; in cross-sectional research the data are collected at a single point in time, in longitudinal or prospective research data are collected at two or more time points moving forward, and in retrospective research the researcher looks backward in time to obtain the desired data. .

 

 

 

 

 

Classifying Nonexperimental Research
by Research Objective

The idea here is that nonexperimental can be conducted for many reasons. The three most common objectives are description, prediction, and explanation.

One type of explanatory research that I want to mention in this lecture is called theoretical modeling or causal modeling or structural equation modeling (those are all synonyms). Causal modeling (i.e., constructing theoretical models and then checking their fit with the data) is commonly used in nonexperimental research.

Here is a way to depict a direct effect: X -----> Y

 

 

Classifying Nonexperimental Research

by Time and Research Objective

So we talked about two key dimensions for classifying nonexperimental research: the time dimension and the research objective dimension. Notice that these two dimensions can be crossed, which forms a 3-by-3 table, which results in 9 types of nonexperimental research.  Here is the resulting Classification Table:

 

 

If the above table seems complicated, then note that all you really have to do is to remember to answer these two questions:

1.   How are your data collected in relation to time (i.e., are the data retrospective, cross-sectional, or longitudinal)?

2.   What is the primary research objective (i.e., description, prediction, or explanation)?

 

Your answers to these two questions will lead you to one of the nine cells shown in the above table.