Chapter 8
Validity of Research Results

(Reminder: Don’t forget to utilize the concept maps and study questions as you study this and the other chapters.)

 

In this chapter we discuss validity issues for quantitative research and for qualitative research.

 

Validity Issues in the Design of Quantitative Research

On page 228 we make a distinction between an extraneous variable and a confounding variable.

 

Statistical Conclusion Validity

Statistical conclusion validity refers to the ability to make an accurate assessment about  whether the independent and dependent variables are related and about the strength of that relationship.  So the two key questions here are 1) Are the variables related? and 2) How strong is the relationship?

 

 

Internal Validity

When I hear the term "internal validity" the word cause always comes into my mind. That's because internal validity is defined as the "approximate validity with which we infer that a relationship between two variables is causal" (Cook and Campbell, 1979. P.37).

 

Types of Causal Relationships

There are two different types of causal relationships: causal description and causal explanation.

 

Criteria for Inferring Causation

There are three main conditions that are always required if you want to make a claim that changes in one variable cause changes in another variable. We call these the three necessary conditions for causality.

 

 

 

 

Threats to Internal Validity
In this section, we discuss several threats to internal validity that have been identified by research methodologists (especially by Campbell and Stanley, 1963).

 

Before discussing the specific threats, I want you to get the basic idea of two weak designs in your head.

 

                          O          X          O

 

In this design, a group is pretested, then a treatment is administered, and then the people are post tested. For example, you could measure your students' understanding of history at the beginning of the term, then you teach them history for the term, and then you measure them again on their understanding of history at the end of the term.

 

 

                        XTreatment          O2
                        ----------------------
                        XControl             O2

 

In this design, there is no pretest, one group gets the treatment and the other group gets no treatment or some different treatment, and both groups are post tested (e.g., you teach two classes history for a quarter and measure their understanding at the end for comparison). Furthermore, the groups are found wherever they already exist (i.e., participants are not randomly assigned to these groups).

 


The first threat to internal validity is called ambiguous temporal precedence.

 

The second threat to internal validity is called the history threat.


The third second threat to internal validity is called maturation. 


If you are following this logic about why these first two threats to internal validity are a problem for the one group design but not for the two group design then you have one of the major points of this chapter. This same logic is going to apply to the next three threats of testing, instrumentation, and regression artifacts.

 

The fourth threat to internal validity is called testing.


The fifth threat to internal validity is called instrumentation.


The sixth threat to internal validity is called regression artifacts (or regression to the mean).


The seventh threat to internal validity is called differential selection.


The eight threat to internal validity is called differential attrition (it is also sometimes called mortality).  Attrition simply refers to participants dropping out of your research study.


The ninth threat to internal validity is actually a set of threats. This set is called additive and interactive effects. 

 

External Validity

External validity has to do with the degree to which the results of a study can be generalized to and across populations of persons, settings, times, outcomes, and treatment variations.

·        A good synonym for external validity is generalizing validity because it always has to do with how well you can generalize research results.

·        The major types of external validity are population validity, ecological validity, temporal validity, temporal validity, treatment variation validity, and outcome validity. I will discuss each of these now...

 

Population Validity
The first type of external validity is called population validity.


Ecological Validity
Ecological validity is present to the degree that a result generalizes across different settings.

 

Temporal Validity
Temporal validity is the extent to which the study results can be generalized across time.

 

Treatment Variation Validity

Treatment variation validity is the degree to which one can generalize the results of the study across variations of the treatment.

 

Outcome Validity

Outcome validity is the degree to which one can generalize the results of a study across different but related dependent variables.

 

Here is a brief summary of external validity:

 

As you can see, all of the forms of external validity concern the degree to which you can make generalizations.

 

 

Construct Representation

Educational researchers must measure or represent many different constructs (e.g., intelligence, ADHD, types of on-line instruction, academic achievement).

 

 

 

 

 

Research Validity in Qualitative Research

Now we shift our attention to qualitative research! If you need a review of qualitative research, review pages 45-48 in Chapter 2 for a quick overview. Also look at the qualitative research article in Appendix B titled "You Don’t Have to Be Sighted to Be a Scientist, Do You? Issues and Outcomes in Science Education.”

 

Now I will briefly discuss the major types of validity in qualitative research, and I will list some very important and effective strategies that can be used to help you obtain high qualitative research validity or trustworthiness.

 

Descriptive validity

Descriptive validity is present to the degree that the account reported by the researcher is accurate and factual.

 

Interpretive validity

Interpretive validity is present to the degree that the researcher accurately portrays the meanings given by the participants to what is being studied.

 

Theoretical validity

Theoretical validity is present to the degree that a theoretical explanation provided by the researcher fits the data.

 

Internal validity

Internal validity is the same as it was for quantitative research. It is the degree to which a researcher is justified in concluding that an observed relationship is causal. It also refers to whether you can conclude that one event caused another event. The issue of causal validity is important if the qualitative researcher is interested in making any tentative statements about cause and effect.

 

External validity

External validity is pretty much the same as it was for quantitative research. That is, it is still the degree to which you can generalize your results to other people, settings, and times.

 

Here is a summary of the strategies used in qualitative research. (Note: they are also used in mixed research and can be used creatively even in quantitative research.)

 


The bottom line of this chapter is this: You should always try to evaluate the research validity of empirical studies before trusting their conclusions. And, if you are conducting research you must use validity strategies if your research is going to be trustworthy and defensible.