Standardized Measurement and Assessment
(For the concept map that goes with this chapter, click
When we measure, we attempt to identify the dimensions,
quantity, capacity, or degree of something.
is formally defined as the act of measuring by assigning symbols or
numbers to something according to a specific set of rules.
Measurement can be categorized by the type of information
that is communicated by the symbols or numbers assigned to the variables of
interest. In particular, there are four levels or types of information are
discussed next in the chapter. They are called the four "scales of
1. Nominal Scale.
This is a nonquantitative measurement scale.
is used to categorize, label, classify, name, or identify variables. It
classifies groups or types.
can be used to label the categories of a nominal variable but the numbers
serve only as markers, not as indicators of amount or quantity (e.g., if
you wanted to, you could mark the categories of the variable called
"gender" with 1=female and 2=male).
examples of nominal level variables are the country you were born in,
college major, personality type, experimental group (e.g., experimental
group or control group).
2. Ordinal Scale.
This level of measurement enables
one to make ordinal judgments (i.e., judgments about rank order).
variable where the levels can be ranked (but you don't know if the
distance between the levels is the same) is an ordinal variable.
examples are order of finish position in a marathon, billboard top 40,
rank in class.
3. Interval Scale.
scale or level of measurement has the characteristics of rank order and equal
intervals (i.e., the distance between adjacent points is the same).
It does not possess an absolute zero point.
examples are Celsius temperature, Fahrenheit temperature, IQ scores.
is the idea of the lack of a true zero point: zero degrees Celsius does
not mean no temperature at all; in a Fahrenheit scale, it is equal to the
freezing point or 32 degrees. Zero degrees in these scales does not mean
zero or no temperature.
4. Ratio Scale.
This is a scale with a true zero point.
also has all of the "lower level" characteristics (i.e., the key
characteristic of each of the lower level scales) of equal intervals
(interval scale), rank order (ordinal scale), and ability to mark a value
with a name (nominal scale).
examples of ratio level scales are number correct, weight, height,
response time, Kelvin temperature, and annual income.
is an example of the presence of a true zero point: If your annual income
is exactly zero dollars then you earned no annual income at all. (You can
buy absolutely nothing with zero dollars.) Zero means zero.
Underlying Testing and Measurement
Before I list the assumptions, note the difference between
testing and assessment. According to the definitions that we use:
is the process of measuring variables by means of devices or procedures
designed to obtain a sample of behavior and
is the gathering and integration of data for the purpose of making an
educational evaluation, accomplished through the use of tools such as
tests, interviews, case studies, behavioral observation, and specially
designed apparatus and measurement procedures.
In this section of the text, we also list the twelve assumptions that Cohen, et
al. Consider basic to testing and assessment:
1. Psychological traits and states exist.
- A trait
is a relatively enduring (i.e., long lasting) characteristic on which
people differ; a state is a less enduring or more transient
characteristic on which people differ.
and states are actually social constructions, but they are real in the
sense that they are useful for classifying and organizing the world, they
can be used to understand and predict behavior, and they refer to
something in the world that we can measure.
2. Psychological traits and states can be quantified
nominal scales, the number is used as a marker. For the other scales, the
numbers become more and more quantitative as you move from ordinal scales
(shows ranking only) to interval scales (shows amount, but lacks a true
zero point) to ratio scales (shows amount or quantity as we usually
understand this concept in mathematics or everyday use of the term).
traits and states measured in education are taken to be at the interval
level of measurement.
3. Various approaches to measuring aspects of the same
thing can be useful.
example, different tests of intelligence tap into somewhat different
aspects of the construct of intelligence.
4. Assessment can provide answers to some of life's
most momentous questions.
is important that the users of assessment tools know when these tools will
provide answers to their questions.
5. Assessment can pinpoint phenomena that require
further attention or study.
example, assessment may identify someone as having dyslexia or low
self-esteem or at-risk for drug use.
6. Various sources of data enrich and are part of the
from several sources usually should be obtained in order to make an
accurate and informed decision. For example, the idea of portfolio
assessment is useful.
7. Various sources of error are always part of the
is no such thing as perfect measurement. All measurement has some error.
defined error as the difference between a person’s true score and
that person’s observed score.
two main types of error are random error (e.g., error due to
transient factors such as being sick or tired) and systematic error
(e.g., error present every time the measurement instrument is used such as
an essay exam being graded by an overly easy grader). (Later when we
discuss reliability and validity, you might note that unreliability is due
to random error and lack of validity is due to systematic error.)
8. Tests and other measurement techniques have
strengths and weaknesses.
is essential that users of tests understand this so that they can use them
appropriately and intelligently.
this chapter, we will be talking about the two major characteristics:
reliability and validity.
9. Test-related behavior predicts non-test-related
goal of testing usually is to predict behavior other than the exact
behaviors required while the exam is being taken.
example, paper-and-pencil achievement tests given to children are used to
say something about their level of achievement.
paper-and-pencil test (also called a self-report test) that is popular in
counseling is the MMPI (i.e., the Minnesota Multiphasic Personality
Inventory). Clients' scores on this test are used as indicators of the
presence or absence of various mental disorders.
point here is that the actual mechanics of measurement (e.g.,
self-reports, behavioral performance, projective) can vary widely and
still provide good measurement of educational, psychological, and other
types of variables.
10. Present-day behavior sampling predicts future
the most important reason for giving tests is to predict future behavior.
provide a sample of present-day behavior. However, this "sample"
is used to predict future behavior.
example, an employment test given by someone in a Personnel Office may be
used as a predictor of future work behavior.
example: the Beck Depression Inventory is used to measure depression and,
importantly, to predict test taker’s future behavior (e.g., are they a
risk to themselves?).
11. Testing and assessment can be conducted in a fair
and unbiased manner.
requires careful construction of test items and testing of the items on
different types of people.
makers always have to be on the alert to make sure tests are fair and
assumption also requires that the test be administered to those types of
people for whom it has been shown to operate properly.
12. Testing and assessment benefit society.
critical decisions are made on the basis of tests (e.g., teacher
competency, employability, presence of a psychological disorder, degree of
teacher satisfactions, degree of student satisfaction, etc.).
tests, the world would be much more unpredictable.
Identifying A Good
Test or Assessment Procedure
As mentioned earlier in the chapter, good measurement us
fundamental for research. If we do not have good measurement then we cannot
have good research. That’s why it’s so important to use testing and assessment
procedures that are characterized by high reliability and high validity.
Overview of Reliability and Validity
As an introduction to reliability and validity and how they
are related, note the following:
refers to the consistency or stability of test scores
refers to the accuracy of the inferences or interpretations we make from
is a necessary but not sufficient condition for validity (i.e., if you are
going to have validity, you must have reliability but reliability in and
of itself is not enough to ensure validity.
you weigh 125 pounds. If you weigh yourself five times and get 135, 134,
134, 135, 136 then your scales are reliable but not valid. The scores were
consistent but wrong! Again, you want your scales to be both reliable and
Reliability refers to consistency or stability. In
psychological and educational testing, it refers to the consistency or
stability of the scores that we get from a test or assessment procedure.
is usually determined using a correlation coefficient (it is called a reliability
coefficient in this context).
(from chapter two) that a correlation coefficient is a measure of
relationship that varies from -1 to 0 to 1 and the farther the number is
from zero, the stronger the correlation. For example, minus one
(-1.00) indicates a perfect negative correlation, zero indicates no
correlation at all, and positive one (+1.00) indicates a perfect positive
correlation. Regarding strength, -.85 is stronger than +.55, and +.75 is
stronger than +.35. When you have a negative correlation, the variables
move in opposite directions (e.g., poor diet and life expectancy); when
you have a positive correlation, the variables move in the same direction
(e.g., education and income).
looking at reliability coefficients we are interested in the values
ranging from 0 to 1; that is, we are only interested in positive
correlations. Note that zero means no reliability, and +1.00 means perfect
coefficients of .70 or higher are generally considered to be acceptable
for research purposes. Reliability coefficients of .90 or higher are
needed to make decisions that have impacts on people's lives (e.g., the
clinical uses of tests).
is empirically determined; that is, we must check the reliability of test
scores with specific sets of people. That is, we must obtain the
reliability coefficients of interest to us.
There are four primary ways to measure reliability.
The first type of reliability is called test-retest
This refers to the consistency of test scores over
It is measured by correlating the test scores obtained
at one point in time with the test scores obtained at a later point in time for
a group of people.
A primary issue is identifying the appropriate time
interval between the two testing occasions.
The longer the time interval between the two testing
occasions, the lower the reliability coefficient tends to be.
The second type of reliability is called equivalent
refers to the consistency of test scores obtained on two equivalent forms
of a test designed to measure the same thing.
is measured by correlating the scores obtained by giving two forms of the
same test to a group of people.
success of this method hinges on the equivalence of the two forms of the
The third type of reliability is called internal
refers to the consistency with which the items on a test measure a single
consistency reliability only requires one administration of the test,
which makes it a very convenient form of reliability.
type of internal consistency reliability is split-half reliability,
which involves splitting a test into two equivalent halves and checking
the consistency of the scores obtained from the two halves.
measure of internal consistency that we emphasize in the chapter is coefficient
alpha. (It is also sometimes called Cronbach’s alpha.) The beauty of
coefficient alpha is that it is readily provided by statistical analysis
packages and it can be used when test items are quantitative and when they
are dichotomous (as in right or wrong).
use coefficient alpha when they want an estimate of the reliability of a
homogeneous test (i.e., a test that measures only one construct or trait)
or an estimate of the reliability of each dimension on a multidimensional
test. You will see it commonly reported in empirical research articles.
alpha will be high (e.g., greater than .70) when the items on a test are
correlated with one another. But note that the number of items also
affects the strength of coefficient alpha (i.e., the more items you have
on a test, the higher coefficient alpha will be). This latter point is
important because it shows that it is possible to get a large alpha
coefficient even when the items are not very homogeneous or internally
The fourth and last major type of reliability is called
Reliability refers to the consistency or degree of agreement between two
or more scorers, judges, or raters.
could have two judges rate one set of papers. Then you would just
correlate their two sets of ratings to obtain the inter-scorer reliability
coefficient, showing the consistency of the two judges’ ratings.
Validity refers to the accuracy of the inferences,
interpretations, or actions made on the basis of test scores.
speaking, it is incorrect to say that a test is valid or
invalid. It is the interpretations and actions taken based on the
test scores that are valid or invalid.
of the ways of collecting validity evidence are really forms of what used
to be called construct validity. All that means is that in testing and
assessment, we are always measuring something (e.g., IQ, gender,
age, depression, self-efficacy).
Validation refers to gathering evidence supporting
some inference made on the basis of test scores.
There are three main methods of collecting validity
1. Evidence Based
Content-related evidence is based on a judgment of
the degree to which the items, tasks, or questions on a test adequately represent
the domain of interest. Expert judgment is used to provide evidence of content
To make a decision about content-related evidence, you
should try to answer these three questions:
the items appear to represent the thing you are trying to measure?
the set of items underrepresent the construct’s content (i.e., have you
excluded any important content areas or topics)?
any of the items represent something other than what you are trying to
measure (i.e., have you included any irrelevant items)?
2. Evidence Based
on Internal Structure
Some tests are designed to measure one general construct,
but other tests are designed to measure several components or dimensions of a
construct. For example, the Rosenberg Self-Esteem Scale is a 10 item scale
designed to measure the construct of global self-esteem. In contrast, the
Harter Self-Esteem Scale is designed to measure global self-esteem as well as
several separate dimensions of self-esteem.
use of the statistical technique called factor analysis tells you
the number of dimensions (i.e., factors) that are present. That is, it
tells you whether a test is unidimensional (just measures one factor) or
multidimensional (i.e., measures two or more dimensions).
you examine the internal structure of a test, you can also obtain a
measure of test homogeneity (i.e., how well the different items
measure the construct or trait).
two primary indices of homogeneity are the item-to-total correlation
(i.e., correlate each item with the total test score) and coefficient
alpha (discussed earlier under reliability).
3. Evidence Based
on Relations to Other Variables
This form of evidence is obtained by relating your test
scores with one or more relevant criteria. A criterion is the standard
or benchmark that you want to predict accurately on the basis of the test
scores. Note that when using correlation coefficients for validity evidence we
call them validity coefficients.
There are several different kinds of relevant validity
evidence based on relations to other variables.
The first is called criterion-related evidence which
is validity evidence based on the extent to which scores from a test can be
used to predict or infer performance on some criterion such as a test or future
performance. Here are the two types of criterion-related evidence:
evidence—validity evidence based on the relationship between test
scores and criterion scores obtained at the same time.
evidence—validity evidence based on the relationship between test scores
collected at one point in time and criterion scores obtained at a later
Here are three more types of validity evidence researchers
evidence—validity evidence based on the relationship between the focal
test scores and independent measures of the same construct. The idea is
that you want your test (that your are trying to validate) to strongly
correlate with other measures of the same thing.
evidence—evidence that the scores on your focal test are not highly
related to the scores from other tests that are designed to measure
theoretically different constructs. This kind of evidence shows that your
test is not a measure of those other things (i.e., other constructs).
the ideas of convergent and divergent evidence together, the point is that
to show that a new test measures what it is supposed to measure, you want
it to correlate with other measures of that construct (convergent
evidence) but you also want it NOT to correlate strongly with measures of other
things (divergent evidence). You want your test to overlap with similar
tests and to diverge from tests of different things. In short, both
convergent and divergent evidence are desirable.
groups evidence is also useful in demonstrating validity. This is evidence
that groups that are known to differ on the construct do differ on the
test in the hypothesized direction. For example, if you develop a test of
gender roles, you would hypothesize that females will score higher on
femininity and males will score higher on masculinity. Then you would test
this hypothesis to see if you have evidence of validity.
Now, to summarize these three major methods for obtaining
evidence of validity, look again at Table 5.6 (also shown below).
Please note that, if you think we have spent a lot of time
on validity and measurement, the reason is because validity is so important in
empirical research. Remember, without good measurement we end up with GIGO
(garbage in, garbage out).
and Validity Information
You must be careful when interpreting the reliability and
validity evidence provided with standardized tests and in empirical research
standardized tests, the reported validity and reliability data are
typically based on a norming group (which is an actual group of people).
If the people with which you intend to use a test are very different from
those in the norming group, then the validity and reliability evidence
provided with the test become
questionable. Remember that what you need to know is whether a test
will work with the people in your classroom or in your research study.
reading journal articles, you should view an article positively to the
degree that the researchers provide reliability and validity evidence for
the measures that they use. Two related questions to ask when reading and
evaluating an empirical research article are “It this research study based on good measurement?” and “Do
I believe that these researchers used good measures?” If the answers are
yes, then give the article high marks for measurement. If the answers are
no, then you should invoke the GIGO principle (garbage in, garbage out).
Three primary types of educational and psychological tests
are discussed in your textbook: intelligence tests, personality tests, and
educational assessment tests.
1) Intelligence Tests
Intelligence has many definitions
because a single prototype does not exist. Although far from being a perfect
definition, here is our definition: intelligence is the ability to think
abstractly and to learn readily from experience.
the construct of intelligence is hard to define, it still has utility
because it can be measured and it is related to many other constructs.
For some examples of intelligence tests, click
2) Personality Tests.
Personality is a construct similar
to intelligence in that a single prototype does not exist. Here is our
definition: personality is the relatively permanent patterns that
characterize and can be use to classify individuals.
personality tests are self-report measures. A self-report measure
is a test-taking method in which the participants check or rate the degree
to which various characteristics are descriptive of themselves.
measures of personality are also used. A performance measure is a
test-taking method in which the participants perform some real-life
behavior that is observed by the researcher.
has also been measured with projective tests. A projective test is
a test-taking method in which the participants provide responses to
ambiguous stimuli. The test administrator searches for patterns on
participants’ responses. Projective tests tend to be quite difficult to
interpret and are not commonly used in quantitative research.
For some examples of personality tests, click
3) Educational Assessment Tests.
There are four subtypes of
educational assessment tests:
--These are typically screening
tests because the predictive validity of many of these tests is weak.
--These are designed to measure the
degree of learning that has taken place after a
person has been exposed to a
specific learning experience. They can be teacher
constructed or standardized tests.
For some examples of achievement
--These focus on information
acquired through the informal learning that goes
on in life.
--They are often used to predict future performance whereas achievement tests
are used to measure current performance.
--These tests are used to identify the locus of academic difficulties
Sources of Information about Tests
The two most important main sources of information about
tests are the Mental Measurements Yearbook (MMY) and Tests in Print (TIP). Some
additional sources are provided in Table 5.7. Also, here are some useful
internet links (from Table 5.8):