Andrew Ho: 2016 Psychometrics mini course - Part 1
Many courses here we have
students from 2 classes.
For tool we are very
pleased to have joining us
to is a professor at the Harvard Graduate
School Education cyclists in the real was.
Who is going to be talking with us
today give you a flavor for some of.
The really important issues
in education measurement.
First thought only measurement folks
in education research to pull.
Lists as well.
Goal here is not to have everything
that one would learn in a foursome.
Sequence in psychometrics but to give
you a flavor of what's out there and
help give you some ground.
To air into is.
Academic here member of the now for
Assessment Governing Board.
He's a former.
Master since you just pick from.
It's a really very pleased to have.
Your Thank you thank you.
We had a game against Michigan.
I should I shouldn't started with the
crowd turned against me so thank you for
this opportunity these are these
are weird things right like 3 hours
measurement What can I accomplish
I'm like actually not sure I think
the presentation is kind of a mess but
maybe deliberately So what I wanted
to sort of leave with like a couple of
provocations a couple of references and
the sort of nagging idea that you need to
learn more and that there are places for
you to do so and so I should start by
saying one of the places you can do so
is right here you have a stealth
psychometricians in Brian Not to mention
a psycho magician in that not to
mention people who know statistics and
measurement broadly likes you and Chris so
you do have a bunch of people who I would
love to have the sitting
in the audience and
watching teach this in their own way and
I would learn a ton from it so
we all would teach this were different in
different ways and Brian and I actually
recently had an exchange about measurement
he was talking to a condom it's and
I was reflecting on how he presented it
and I was a ton of fun and so I wish you
know it's actually I was talking to Matt
about this earlier too I wish we could
talk more about how we teach this and this
is I think it's a great opportunity for
me to engage with you but also to get
a little bit from from people about
the different ways that they approach this
because we could learn a lot from them.
So here are my provocations So I kind
of want to do these like you know these
listicle ce writing this is like how to
get people's attention 7 things you need
to know about measurement but I'm caving
I said that I'm going to do it anyway so
here are here are those those provocations
I'm being deliberately kind of
extreme here just kind of
to procure a little bit so
validate score uses of the question
I find myself asking my students most
often when they are starting to conceive
their measurement projects is what for
and not just what for but what and
like what is a score ultimately whether
it's a scale score or an average of scores
or regression coefficient ultimately
those scores are what's used and
we don't validate tests we validate
those ultimate uses of scores
to content is king not models content so
if you start analyzing data ever
without knowing what items there were
you should slap yourself on the wrist or
feel your advisor slapping yourself on
the wrist What are you measuring and
you need to have an embodied
experience of what that
is you need to put yourself in
the shoes of your participants and
feel what it's like to be a part
of that measurement procedure
have a tendency in measurement to jump at
the neatest hottest sounding models
with the longest acronyms stop it start
with simple simple descriptive statistics
in measurement and I think of classical
test area as the descriptive statistics of
measurement you should always start there
in the same way that those of us
data start off with what command.
Summarise so did so so
Alpha is your summarize of measurement OK.
this is like this these are not the droids
you seek or whatever it is go to the
reliable the that is most often calculated
is probably not the reliability
that you're interested in and
I'd like you to leave here today being
able to answer your aunt or uncle when
they ask you what is reliability I want
you to be able to answer that question I
remember when I was getting my masters in
statistics on them because I don't agree
what the standard deviation can you
think of how to answer that question to
your uncle as I could give you an equation
like how do you actually describe that in
a meaningful way I think that's
reliability like the standard deviation of
measurement be able to what is point
that mean and you should be able to answer
that I mean today if you can't read.
I don't respond there is just a model.
We always hear we should really
have someone who knows and
in response to it I wish I had someone who
knows I know response theory which I so
many could teach I'm response
theory is just a model and
is also a very very useful model so
I'm both going to demystify it and
focus you on what it does particularly
well 6 your scale is pliable bend
it don't break it and this is something
that Brian wrote about recently as well.
The numbers that you should you should
think of them as like solid ground like
the distances that you sort of see they
can there's kind of like this is kind of
like this bridge it's like springs between
the boards maybe like there's like a sense
that the scale is pliable to shift
a ball but not breakable and
there's actually empirical ways
we can address this tendency and
your or the judgments you make based
on your scale information should
should be robust to that spring in us.
And then 7 again this is a reiteration
of other things know the process that
generated your scores and
use them accordingly do not go beyond what
the data suggests that's just a general
recommendation so these are my
publications they sound just like
floating talking points now they're deeply
embodied to me and I'm going to try to
do my best to make them feel meaningful
to you over the next couple of hours.
So but I want to start by just stepping
back and again when you think measured
what kind of resources can
you use moving forward and
in the e-mail that I think some of
you received from me you've got $2.00
to $3.00 citations right 2 references
in the 1st Are these the standards for
educational and psychological testing
what we're doing has has decades
centuries of history and a field has built
up around it that has some guidance for
you right and 3 the major body is the
American Educational Research Association
the American Psychological Association and
the National Council on measurement and
education came together and
actually agreed on something right these
are the standards of the field and this is
a very powerful tome not just for you but
for the people who you're developing
tests for you can say I did this and
these are the standards of the field
you should have this book and
there's a discount for
members of this of any of these but
like if I were to recommend one thing
these are the authoritative standards of
the field they're not perfect I've
got a kind of quibbles with them but
they're powerful and
it reads actually is as a good as a as
a pretty reasonable like intro text.
this because 160 bucks and
I remember being a student and
it's also very much a reference text but
this is sort of the cut bible of the field
has all the sort of heavy hitters
in measurement who contributed chapters
to it it is sort of the authority to
like what we would cite when we
say if you had generic a site for
reliability you would go to
Hartle 2006 generic site for
validation you would go to Cain 2006 and
those are the 1st 2 chapters in this book
do not buy it unless you
are really into the stuff.
But it but have it on reference of it's on
reference for easy reference your library
so much you put it on reference frame so
they would be my sort of to go to books
for for educational measurement
probably and you should
look to them if something is provoked
to you as places to go for citations.
So how do we learn measurement I think
this is important to visit visit too so
like you know this is very much I think in
the way that the way this is like reflects
the philosophy of teaching here too but
I always want to sort of point to my
fellow by and I'm like eager to look
at other still abide to from folks
other folks who teach math but if you want
other references there they're up on my
website there's like sort of like what
I think of as where to go to for I.R.T.
what I think you know what I think people
should read when it comes to differential
item functioning when I think what people
should read when it comes to standard
setting so they don't forget to look
at people still by when your reference
hunting and you want to say someone
said a cut score who should I cite for
the whole a whole cut
score setting thing and
Philip I can be very useful in addition
to that in addition to this tome
where it's Hamilton and Tony Adams who
did on who did standard setting so
my fellow by and the syllable of others
like Matt are good places to go for
references on these things.
And then again like you know learn
it use it learn it again and
use it again like in
practice makes perfect and
all of your classes your methods classes I
think you're using data getting your hands
dirty struggling with those state error
codes looking at those manuals right so
you know help help help them clicking on
the P.D.F. that's that's what you're going
to be doing a dodgy and I just want want
you to want to recognize that measurement
much like the other methods you're
learning requires that struggle and
that patience so this is something
that's a little trick I just want
to give a shout out to a few people
who contributed to my Google Docs
this is something I do to like incentivize
and encourage out of class reading and
out of class discussion so
there are a bunch of tools for this and D.
has and it has a new thing called to
result at some of my colleagues at Harvard
of develops or just open google docs but
it looks like this this is what I asked
some of you to contribute to I said
Hi I'm students is the typical pre class
discussion that we run it is to see I
ask questions I have you respond to them
by 10 pm last night and then I reply
like I am kind of a night owl and so
like in between 10 pm and Dawn for a class
I like I sort of give people like answers
I have little conversations and sometimes
for the for the people who contribute late
like around 9 pm We get into these
sort of discussions online actually
how did someone who was I know who it was
but I was online with some one of you.
And and just going a little bit back and
forth about how to write out
an equation so so thanks to Josh G.
There you go Josh so so
so we I had a little.
Cross talk with you I'm not sure if you
check back in solid I saw what I wrote but
this is a part of the read write
do this is just sort of like
how to sort of stay engaged.
And here's an actual Doc So
thanks to Josh G.
thanks to Fernando you can
see I'm replying italics here
thanks to Stephanie H.
Stephanie I missed your comment you must.
OK I'll reply later I promise Karin.
So there are a lot of really good.
Some derivations we did here Stacey.
So yeah and then some that there's always
I always sort of leave a space Cassandra
didn't get a chance to reply to you but
Josh here there's a general space for
general questions and discussion and
you asked and good general questions
that you can have time to engage with
towards the end of this class today
I should say to like interrupt.
But I think it's important sometimes
you just might talk straight street
pedagogy and like how to learn this stuff
and how to stay involved so read write do.
So again there's a 7 principles
I'm going to start with and
you start with the beginning and
start straight from validation so
we don't validate test to be validate
score uses to talk a little bit about
the validity theory and
this might seem a little bit detached and
I'm going to get more technical later on
I know I feel like this is very talking
about this as tribal I'll try modal
audience potentially So I think I might
interest some of you sometimes and
others of you other times but
all of this is important in belongs to
the body of measurement so I hope you'll
survive remember even the things that may
sound theoretical even the things that may
sound too technical but are all in
a continuum and think of it as what we do.
So validation This is the more recent
depiction of what I think of as
the standard for validation in educational
measurement in particular I contrast that
with Matt who teaches more from the the
psychological measurement paradigm which I
think has a slightly different perspective
on validity but but Michael and
the field is an educational measurement
in particular is very utilitarian
very instrumentalist we
care about the ultimate use
right it's almost a theoretical if you say
if you if you take it to a certain extent
we just don't even care about the numbers
as long as the interpretation or use
is correct that's extreme but that shows
you what that what they're emphasizing
here what we're emphasizing here to
Val to validate an interpretation or
use of test scores is to evaluate
the plausibility of the claims based on
the scores as an argument based approach
you are building an argument with evidence
over time there is never a point
where something is valid it is part
of an ongoing evidence building process
and that is deeply unsatisfying
right wouldn't it be great if there
were a correlation coefficient and
once it exceeded point 7 you said Jack and
this is a super frustrating
like call to you to never do
that right and I think it's particularly
frustrating for introductory students for
whom like that they might who even for
those who like might not have careers and
measurement to say they really have to
do all this and I guess I'd say like
actually no you don't really have to do
this but at least you have to know that
these are the sort of standards of the
field even if you selectively ignore them.
So and again this is just these are for
measurement there is my colleague
Derek Briggs who kind of disagrees with
this utilitarian instrumentalist
current status quo in measurement but
there actually debates about this in the
field right what is the validity you can
write a paper on that right and contribute
to the discussion of what it means for
the use and interpretation of test
scores to be valid and appropriate and
we broached this this morning when I was
talking about the use of this new data
set that my colleagues and I have created
that allows you to compare districts or
school districts across states and we're
not we don't just ask is that valid or not
we as we say is there are the uses of that
are the interpretations of that valid or
not and there's some really good
feedback from the faculty members and
students in the room about which
which research designs and
which research inferences would be or
would not be appropriate in those
situations is very similar what you're
trying to do with the scores and
is that appropriate is that supported
by the evidence so I would say these
are there these are different
definitions of sort of validity or
full of schools of thought about validity
and not reading all this text because
I'm sort of that sort of leaving
these slides as a reference but
we are in a very instrumentalist
even utilitarian moment and
educational measurement where we care
ultimately about how you're using those
scores not about the test or
even the construct is about the score use.
So again modern test validation theory
is dominated by instrumentalists I'm
concerned with test uses an interpretation
and I'm acknowledging that this can
be frustrating because it kind of takes
the control away from your special little
instrument and it's in its ultimate
scores and places it in this fuzzy domain
where people pick them up and use them and
you might kind of be responsible for that.
So I think a validity and
as I say as I tweeted before it's like
I'm not ashamed to use mnemonics and
so I think of 5 sources of Liberty
evidence and I call them the 5 seas so
the 1st is content right so the 1st
to take the test what is it measuring
There is a good overview of alignment
of 4 big testing enterprises
to the Common Core recently Morgan pull it
off and then see DURIE publish this piece
in Fordham earlier this year which is
which is basically a content study right
do park and smarter balance these big
testing consortia and as well as M.
CAS and the A C.
aspire these the Massachusetts state test
and a.c.t do they aligned to the Common
Core state standards this is a content
study and there I think there are too few
people frankly delving into this like
arena which is currently
sort of dominated I think by
more model based statistically based
approaches so I'm just sort of reiterating
that content is important serious
important cognition is another source to
go as another source of evidence
is like when you take that scale.
Are you thinking what the designer
intended you to be thinking as I'm
thinking through this math test
as I'm thinking about whether or
not I'm greedy or not think about the
studies of great recently that have been
concerned about reference bias right
that is to say like do I feel greedy and
can you compare it across courses or
my referencing my grit to the people who
happen to be in the school or in this
classroom right so how are people thinking
about it cognitively the way we we have
seen they could the evidence we can get up
often comes from Sir think aloud
protocols as well as a parable analysis.
Coherence is where the field since it
seems sort of stuck with validity and
there aren't a lot of what I'm going to
talk about subsequently is going to be in
this into this 3rd seed so this is where
reliability analyses come up if a C.F.A.
I.R.T. this is what not teaches as
well as well as me this is I think
what people sort of assume measurement is
from a technical standpoint and what I'm
highlighting here is it's only one city
right you've got to think about content
you've got to think but cognition and sure
you can do your reliability analyses but
that's only a piece of the puzzle another
piece of the puzzle that is often this
comes up a lot in structural equation
modeling comes up a lot in economics too
where you're trying to predict future
outcomes does this predict college
attendance graduation or
college entry or freshman G.P.A.
or future outcomes or more concurrently
does this does this correlate or
not correlate with things
that should be similar and
things should be different you
sometimes hear this is convergent or
discriminant ability but this is
again only a piece of the puzzle and
the Fitzy is consequences right evidence
based on the consequences of testing
you could think about this even as
a counterfactual like had I not undertaken
this measurement enterprise at all but
would have been the difference so
doesn't think about the scores as
much as the use of the scores and
like that has has the act of testing and
measuring itself had some consequence and
so this is a fairly controversial
relatively recent addition to
the sort of the Litany framework but
these 5 sources of Lady evidence are
clearly articulated in the standards and
what you should think of when you're
designing a measure when you're using
a measure as the kinds of
evidence you can live.
So so this is sort of in contrast
with what with what I think of when I
think people are thinking of validation
commonly I developed a scale with good
theory I fit a C.F.A. and got good can for
confirmatory fit index and
my reliability is greater than point my
scores predict desirable outcomes so I
have a valid reliable measure that's like
the common sort of articulation of like
a good baseline study about I'm setting
that up as you know so that's content
that's coherence that's coherence to this
is correlation and that's incomplete or
sort of missing cognition room if we're
missing consequence you're missing this
argument for use what are your scores how
we use them what would have happened had
you not measured and so
these are other questions you could ask
just with complete this sort of validity
framework so it's more than just.
A good fit index and
good item parameter estimates.
So again 7 key principles we don't
validate tests we validate score uses
That's what I was covering and I want
to emphasize content a little bit and
then dig into a little bit
of classical test ary And
I think that'll probably take
us to the break or thereabouts.
then we'll get into the reliability and
I are to be sure of after so
this yes sure.
Yeah talking about consequences how it
should be in the context of what you
mean like if if a student had never
been tested then what other measure
to measure look underlying ability to or
think that we're getting
the score ultimately is is yeah is used
for something right so once once we
test what's the sort of theory of that
making a difference in some way and
it could be like publishing an article and
having that feedback into the system it
can be very abstract in that way it
could also be the teacher is going
to use it to give you feedback and is
that feedback going to have a positive or
negative impact on you right or it's going
to lead to a value added estimate for
a teacher and they're going to respond
differently to teach so it's like Had
that not happened a whole
process not just the OR
it like you know the score but the use
of the score in this theory of action
had that not happened what would be
the difference so I think that's kind of
a pretty gold standard level of
like I mean we're taught but
a major evaluation at that point but which
is why this is sort of a controversy all
sorts of related evidence
because like good luck and
how long do you wait for
long term outcomes but but this.
I mean from an economic You can be
because of the kind of catch all the time
you know you have people.
Like how can you think of them.
So again so you know in all the ways
that I think you're trained to write
as economists right so I think you again
like and I didn't I wasn't being glib and
I was sort of saying this is like why
we're glad we have people like you is
because I think you are asking like what
you know what is the counterfactual for
you know if we didn't have high stakes
test based accountability like we'd
have some sort of paper by some guy named
Brian Jacob and Condi or something and and
and sort of think about what happened had
there not been this rise in accountability
at this particular time so these
are the kinds of evaluations I think that
I'm not soley putting this in the in
the in like in economics like that but
that that said I do think that's my
encouragement to you is to never just
think of a test as something
that's validated up in the air but
as like part of the results in the score
that is used for a purpose and
if that purpose is for you to publish and
get some correlation coefficient and
get in a journal and that's great and
that's part of your theory of action and
that's pretty light but all but
ultimately I sort of say like but
you know why are you doing this and
that's why I'm sort of for
pushing people to go is that ultimately
your scores are used by people for
something can you can you
describe that to me please and
that's what I find myself asking most
students like that's what's missing and
when they say I want to
create a measure of X.
I'm sure like why you know those scores
we're going to do with them
what's going to happen and
that's what that's often what I find
missing in their their thought process.
Thank you we're here and I'm trying
to figure out correlation you said
evidence based on relation
to other variables and
so I'm wondering if by that you mean like
I would validate one standardized test by
its relationship to student scores
on a similar kind of test of similar
kind of content or reading of
things much broader than I'd like.
This chance to and how the critics
like high school graduation you're
going to college and so
how would I know those kinds of things
before like if I'm using these as
a foundation for measurement and
developing I haven't given it yet so
how do I have evidence on this is.
So so this is why crown Bach and
all the sort of.
People who have developed validity
theory over time have been.
Very clear that it is an ongoing
process that it's not I mean again and
this is where psycho magicians struggle
with dealing with the outside world
because the outside world is like show me
your valid measure and you're like but
this is this process that takes a look
at Show me your valid measure and and so
it can be frustrating but
this is how the field thinks about it I
think you have to wear different hats and
when you're talking to people who have
that their definition of liturgy and
just say this checks all the technical
boxes and you do want to at least some
correlations with concurrent out
concurrent variables in some way but but
look at the end cast Tech Report technical
report look at the report here for
your tech your deep What is it now and and
you'll see that the all of these are laid
out in there in varying degrees of
depth and usually coherence is a massive
section with classical test theory I or
T differential item functioning alike and
correlation to small
consequences is a paragraph
cognition is like we did a lab and
content is very very fleshed out with
content frameworks and the like so
this is why I explicitly walk
through Technical Manual You know
when you finish my class you should be
able to read a technical manual for
a state testing program whose data
you're going to use and figure out
what implications it has very for your own
analysis yes that's a good model to check.
What are some of.
The valid for the test but for.
What are some of the kind of.
Thing and I'm wondering when you
were talking about federalism focus
you seem the one to see complex necessary
if it was really going to meet that.
Geared up on care I don't care what
I don't hear the reliability of
its core how well he learned
in college now that.
You're going to be
anything other than for.
This is a good question so
this is where the economy is so
we're probably shouldn't like over and
over going to miss dinner over drinks at
some other point we will have a detailed
argument about or debate about why these
things should matter I think I mean so
from a very utilitarian standpoint in the
near term before you get those long term
outcomes you know if you're developing
your own measure you need to stand on
something in the near term before
you've got those long term outcomes.
The here yeah and
it also I think it also I mean
I don't know like if you happen to find
some spurious correlation of something
I mean there's got to be some and you
are interpreting when you completed a C.T.
score that there is some sort of college
readiness and you know when you say like
point 3 It's like socioeconomic
status correlates point 3 and
it's like you don't say are college ready
based on social economic status right and
so the interpretations we use like matter
is the sort of psychometric argument and
so you know when whether I enter Be
specific about that interpretation and
what is the warrant for
that interpretation and
if it's only based on social economic
status and the warrant seems.
Detached from the human So I think this
is a deeper philosophical argument you're
raising that I don't think should be.
So I but I think it's a good one and
certainly some that might that my
students have advocated for and
it's certainly econ leaning.
But you know the you know what I
often fight with is like why do
we care about freshman G.P.A.
I mean look at that's a horrible measure
I kind of wanted to kind
of want freshman G.P.A.
to predict my on my high school test
because that's a better measure because
of the content the directionality I
mean so it's does arise I think from.
The items in the content is
the is psychometric percent.
But so so on to a little bit
of classical test here in
the tools that we use to
evaluate in particular Clarence.
Or and content so this is sort of
like my checklist for it like how to
get into a sort of secondary
analysis of test score data right
you get a bunch of you get a state a D.T.
a file and it's got people in rows and
there are all these items all these like
columns that correspond to items and
I guess you know so this is like my going
to skip around is going to go 12378 or
something like that but this is this is
sort of part of a larger checklist and
again like you know this is from
John will it's presentation as well.
No you're right it's right like read each
one take the test get a sense of what it's
trying to measure.
So so this is an example from a a.
Measure of like self perception of
teaching success you have high standards
of teacher performance you're continually
learning on the job you're successful in
educating your students it's a waste of
time to do your best as a teacher this is
negative negative negative polarity you
look forward to working at your school
how much of the time are you satisfied
with your job right and so this is like my
advice to you is never go into an analysis
without actually looking at the items and
sort of taking that like scoring the test
thinking of yourself as a subject and
then you have all these sort of like your
scale items is one to 6 you see here some
someone snuck in a one to 4 item
this happens from time to time so
do not get caught unawares do not type in
Alpha without recognising that some of
your variables have different items skills
than others because it will give you
incorrect answers so so take control
of your scale and know it backwards and
again I'm going to in the interest of
time I'm going to jump through this
always on the scale of your items
right to score your test
how is it actually being
scored is it a some score it isn't.
Average are you reversing
some of the play or
any of your some of your items are you
stretching the scales of some of them so
the algo from 0 to 100 what do
you how you actually scoring it.
So if you if you look here right again
you're going to want to sort of what I
recommend that you do when you're actually
going through this is reverse it yourself
like take control in state and reverse
coat it so that they're all pointing
in the same director because and
then make this because otherwise
I have I found myself making mistakes
is some very practical advice for
you to not slip up in the sort of data
in the early stages of an analysis
so you know again look at your data get a
sense of the missing this label your items
make absolutely sure your items skills
are oriented in the same direction or
you're using code that
recognizes when they're not
positive should mean something
similar if not fix it.
Here's more exploring I have mandate
that people always like give me discreet
histograms for items scales I want to
know Mike how many ones there are how
many to 0 how many threes fours fives and
sixes I want to see if you've got a 7
point Likert scale if no one is picking 6
or 7 ever I expect you to know that from
the very beginning and don't start running
I.R.T. until you have a sense of your.
Data actually look like.
This is important as well does
a one mean one at all times it is
is it always like strongly disagree
when you have a scale that goes
like one to 4 right so if I have
strongly disagreed strongly agree and
then I have not successful it's a very
successful and this is one to 6 and
this is one to 4 and I throw that
into alpha if I throw that into like
a reliability analysis what is
a going to do is going to assume
that very successful means slightly
agree does that make sense.
It could make sense you better think
about it and make a decision so if so
the idea here is that all of
these items scales are not
in a classical analysis are are they
think of ones as ones and
sixes sixes so you better take control of
that and make sure that that's right so
often what that entails is 2 things
one stretching this 124281 to 6
or actually just forcing this to be
one forcing this to be 6 forth and
forcing this to be what 2 and
like actually equally spacing that item
out so that you're saying not successful
is like strongly disagree very successful
as like strongly agree so one of the big
mistakes I see people making when they
get the scale is a secondary data analyst
that assuming that all items
are sort of interchangeable and
that the player he doesn't matter and
you sort of control over that.
Another way to approach it is to
standardize within each item so
what you're doing is you're to
your set you're just dividing by
the standard deviation unit in each time
and each and each item and in that case
you're saying that strongly disagree here
and strongly disagree there might not mean
the same thing depending on the variance
of each of those ITEM ITEM distribution.
And that's weird too like when your
liquids like or scale items are all
strongly disagree to strongly agree do
not standardize right because strongly
disagree means the same thing across those
items and if you standardise you lose that
information Similarly if you have an
educational test that has like correct or
not correct should you standardize
absolutely not correct is correct and
the same thing so do not standardize
you know in those cases either as these
are the like the little things that seem
trivial and I feel like in my in my own
way in my own students like analyses and
I'm not running through there coming up
with absolutely incorrect alpha values
I can even just like the baseline
descriptive statistics let alone getting
to I.R.T. or structural cohesion modeling
or attack so you've got to take control
of your data from the very beginning and
be very very careful and intentional about
every single step that's like general
advice for statistics period right but I'm
saying it still applies to measurement.
OK So this is a baseline reliability
analysis check this out Alpha X.
one to dash X.
as is that should be your template and
the items gives you all these
items to 6 as is I saw I
have this sneaky suspicion that this
is leading to inflation of reliability
coefficients throughout state and users
and perhaps other programs as well but
as is does is it says the direction of
the scale like the direction of the item
scale positive is always positive
like if you coded as positive and
treating it as positive if
you don't include as is
there could be a really bad item in your
scale that correlates negatively with all
the other items negatively and
state a will flip it for you.
Without telling you will show up here but
you might not notice it without telling
you it's going to flip it for you which
is to say you've got such a bad item that
status as it can't possibly be
that bad in reverse it for you and
that's crazy to me that they do that and
so you thought this is that for
a lot of elementary analysts dramatically
over interpreting their simple.
Alpha they're simple reliability
value because they're.
Going To Do you know best but
but but but anyway so
this is be my default code to make sure
that you're controlling it appropriately
be intentional at every
step of your analysis and
know what the direction is and
know what the scale points are OK So
this is I'm going to I'm just going to
short hand wave 3 this but these are.
Various discriminations statistics
they basically are like does this item
correspond to the sum of
other items on the scale
does this item correlate with other
items and this is the coherence question
this is an internal correlation does this
item correlate with other items on a scale
which is really kind of what is at
the heart of classical test theory I or
G structural question modeling
factor analysis and the like.
This is an example of a little bit
of you know more pseudo code from
state A for you.
How many people don't use data.
So and you're using M.
Because this is why we include a whole
bunch of do files and I've sent Bryan
a couple off and I'm more but I'm happy
to give you sort of templates for this.
too we'll talk we'll talk more about that
the simplest of the good cos it will
test every kind of descriptive stats.
To the you know like OK you know.
Anyway what we're worth running so like I
mean they they presume that you sort of
done all that already and so do all that
already like to do that 1st as a as I'm
recommending it as make sure you sort
of have control over your scale.
So again you know coming in as a sort of
content is king there in the sense of like
you know your items know your scale and
get a sense of what it's
trying to measure and
don't just validate it based on whether or
not it predicts life earnings next.
But if it were the debate.
What exactly were they.
Looking at like that.
In the sense not in the sense
of like I mean you want to
read a book on the question
because I want to get.
Like I mentioned.
Some of that question but maybe.
I can see Mollenhauer.
Right so this is this is a subscale
question this comes up all the time so
Alpha is a property of of a of a scale
right and if you want to create subscales
get get information about each of your sub
scales that's what Alpha should be for and
what else if you throw an alpha across all
of the items across subscales it's asking
how coherent is this across subscales So
the question I always ask people who
are using subscales is what's the question
how are you using your scores right so
that you know if you take a cynical
approach from like you know at heart of us
always like if you give policymakers
to numbers a lot and together.
So that you know so
this is like the you know so
that your great scale the Angela Duckworth
a Tim Duckworth and Queen 8 item great
scale there are 2 subs course we
think people are doing with them.
Adding in the getting so if you want
your question my question is what
your question should be what is the
property of the score that is being used.
This is that this is the utilitarian
sort of instrumentalist of you and
if you are creating a scale with like
that people are using those subscales
an evaluative each of them accordingly and
then take alphas for
each of those subscales report outfits for
each of the sub scales I'll show
you how Angela and Patrick.
Do this and
shortly in their actual paper so
yeah so so so which is just to
say good to have subscales but
then then what I would
do is Alpha out C.T.
analyses on the subject and later will
talk confirmatory factor analysis and
all that jazz or actually that well
that's what his class is good at.
Go So this is this is the this is a paper
that I have everyone in my class dig
deeply into this is Angela Duckworth and
and Patrick Quinn's.
Journal of Personality
assessment paper in 2009 that.
I was talking with not about this is
a very common practice to develop
a scale that has way a ad that has now
way too many items but a lot of items and
you might not you might want to think
about how to minister them feasibly
in a flexible situation and so you can
use Costco test area in response to
a response they're both very very good at
figuring out how to shorten that scale
like how to how to preserve information
while while reducing the number of items.
This is a say you know I just gave myself
I just gave you advice I'm trying to
follow it this is sort of a brief
description of the great scale I actually
have my students take this so we can
like analyze their data new ideas and
projects sometimes distract me setbacks
don't discourage me I've been obsessed
with a certain idea but I am a hard worker
I often set a goal but later choose to
pursue it so I'm shortening them a little
bit this is to give you a sense of how
great operationalize So this is their item
scale in this paper they're sort of saying
we had a 12 item scale we're going to 8
it will all be fine don't worry about it.
So part of my screenshots
here see table one for
item level correlations after excluding 28
I'm sure each subscale I talk in subscales
here right there is all things out in
great scale this displayed acceptable
internal consistency that's code for alpha
with alphas ranging from point 73 to point
a look at their table to write
again we spent a lot of time digging
into these articles in class so
this is like you know West Point the
famously her National Spelling Bee sample
Ivy League undergraduates and these are
conduct also values these are the values I
was describing point
the sum that's the total scale that's
the that's the reliability coefficient.
For the overall scale and
then she breaks it down into pursuits of
effort and consistency of interest and so
the question I would ask in this
case is again what's being used and
if you're treating these separately you
can see what their alpha values are and
then if you're treating them as
a whole that that's the that's so
you can sort of cover your
use cases here and say for
those purposes here is your level of
internal consistency that makes sense.
Absolutely and so this is why your
classical test there isa to 6 are your
descriptive statistics your knee jerk 1st
reaction and after that we're going to get
to a more powerful framework that allows
you to answer questions like the ones
who's asking and so this is what I
consider level one this like summarize and
I really do mean that is like the very
after that you get to more
sophisticated questions OK so
by the way the what I always have one
of my questions my google doc questions
is is kind of this annoying I guess
what I'm thinking questions but
it's like Does anything look off to you
about this and I'm just going to sort of
this is like a tough question so I'm just
going to pause and and the just take
a look at this table in particular these
alphas these alphas compared to these
alphas and I just so this is you know
going for items for items and 8 items and
I just want to sort of this is to have you
take a look at that and just get a new
curve gut reactions as to what
I find a little surprising.
There's a bit of it that.
I have a plan to in the audience.
There can be a couple answers here so
don't be shy.
For example the man.
Who does point 73 or
an 8 item scale I have that's wacko.
Right and so I'm not sure if he's
correcting for that and didn't mention but
or if there's something weird going
on in the sub scale relationships but
that is not what you expect what you
expect when you have many more items in
fact we're going to show you a prophecy
formula that predicts this when you
have more items in the same way that you
average over more things you have center
deviation over route and is your position
the more you average over the more
precision you have now it is a little
surprising that it's accurate that's
a kind of discipline perception that
you'll develop with with with measurement.
Cause that a lot but
Joining me to go from this.
Which is that much that.
You were to be purely So this is one
way right so we're going to develop
even better ways with I.R.T. But this is
just sort of a ranking of how each item
correlates this is the item rest
correlation is a literally the Pearson
correlation a simple vanilla correlation
between an item in one column and
the sum score of all the other
items in the other column so
this is a measure a very descriptive
statistic again kneejerk summarize level
descriptive statistic of how well this
coheres with the rest of the scale
we're going to see a better version
of this is going to get to higher T.
but often they very rarely tell
dramatically different stories so
this is why again we sort of start with
our feet on the ground with a basic
analysis and then get advanced and
I are today.
So you so wish to ask your question if you
were to be purely cynical about it and
didn't care at all about content
you drop you drop maybe one in 3
you know rerun the item S.
correlations maybe drop a couple more if
you felt like it and then calculate Alpha
for whatever's remaining and I don't
recommend you do that because content is
king that's the be careful of throwing
away a subscale you care about but
and imagine that an educational test where
suddenly you're not measuring mass or
something right so you can imagine
that that would be dangerous but but
but that's from a purely statistical
standpoint that's what you could pull off.
So so good we're going to great I'm going
to judge do I have this Yes Skip to
slides on classical test or E.C.L.
that's that's it sort of there for
you there's a bunch of equations there
sort of putting that in as like stuff for
future reference what I want to
do is talk a little bit about
why classical test theory is a theory and
what it predicts and
why it seems like it's useful so what is
classical test theory actually predict and
why do we think of it as theory the 1st
you know what can we infer from classical
test theory 1st variation
increased reliability and so
this is akin to the logic you are using
might sort of flip it on its head that you
know if you if you ask what the
reliability of a grade 3 of a set of Grade
ask for the reliability of Grade 3 grade 4
grade 5 scores is together you're going
to get like a point $85.00 right so
as you increase the variance right in the
same way that we you know as we know from
correlations period reliabilities or just
correlations I forgot to forgot to ask
what's reliability We'll get to that
again but just like any other correlation
as you increase the variance
you increase the scatterplot
rate you increase the sense of correlation
and you know whiteboard in here.
Later is that.
So I know that.
Because I don't work wow OK that the mind
OK I'll get the I'll get that shortly.
Go see Thank you.
So this is a a read derivation of.
The liability if you square
both sides put air put X.
under air that's the proportion
of air variance and
then one minus that is the proportion
of true score variance.
So that that's reliability and so if you
do a little bit of algebra here you get
this expression and so
in terms of the observed set you can
you can derive the senator measurement
in terms of the observed as the observed
standard deviation and reliability and
as you increase that standard deviation
you you get you get you get you're
going to increase your reliability
in the same way that I'm going
to draw right now so thank you.
So this is.
So like let's think about.
Which I think if you're
just this is a Grade 3 X.
Are grade 3 X.
prime or something like that so
let's imagine these are replications
of procedures or grade.
So so in any case like if you have some
correlation that's like Grade 4 but
then increase the scale and
have a grade 5 here and Grade 6 here.
As you keep going up the scale so
Grade 3 here.
So if you look at this sort of scatter
plot Harry like add that correlates around
like point 6 or so but as you can as you
can see as you keep sort of Caterpillar
ing this out is like a caterpillar I
know it's not the greatest picture but
the idea is that ideas that now hey this
correlation looks more like point $8.00
And so the greater the greater
the variation you have
the more the more reliability
you'll have so I'll say to you so
one of my students will is doing a pretty
neat project with Dana McCoy She's
using Google Street View to rate
schools and like the sort of perception
of like school quality from what you can
tell in Google Street View and she kind of
she made a mistake upon reflection you
know when thinking about this prediction
of predicting of taking schools that
were too similar to each other and
they're like the stick let's take a bunch
of schools are too similar in quality and
then look at inner rater liability and
item or liability across those schools
upon reflection which she should
have done in order to like in
scale development is to make sure that the
variation very deliberately was reflective
of the variation in the population so that
she can get a reliability that corresponds
to that that said classical test or
he does give you a tool for for.
For correction correcting for
the variance in the sample you have
versus variance of the population you have
versus the variance in the population you
ultimately care about so
this is like your general expression for
how like the changes in variation will
increase the ultimate reliability and
I'm just again putting
this here as a reference.
So that's a again a very a classic
thing you should know about
correlation is that as the variance of
the true variance increase in
the population it will increase Yeah.
We're going to think this is.
A distraction the root of this is kind
of a random subset of the population.
Were battle weary but it's been a gamble
and I think I've seen some educated.
Looking at it said only that the size and
The college we're looking at the.
Trial Court but the.
College or inappropriate is.
The whole the.
Whole array of the giving
some of the work to lay.
There really but the.
Of the so I should've had this been
a 12 week course in measurement I would
have made sure to hammer that home
repeated so the classic example is for
example the correlation between like
a city scores and freshman G.P.A.
at the University of Michigan right and
that tells you what it tells you but
if you're interested in Had they had
everyone been a minute what with
the correlation have been that would have
been that would have you would have seen
that would've been larger but
you can't tell for the reasons that
that Brian Brian suggested I should
add here here's my general advice
if you ever were to undertake this because
if I were a reviewer I would I would
then you if you didn't follow it and that
support both right report both the initial
correlation and the you know as you
assume there's going to be attenuated or
discipline you wait a correlation and
state your assumptions clearly but
never just say and
here's my just attenuated correlation and
I actually reported this a tenured
correlations in my presentation today but
in the paper we report both says
I'm trying to follow my own advice.
So similarly that advice is going
to is going to hold here as well
if we're ever going to talk about standard
deviations the standard deviations
observe standard deviations are inflated
due to measurement error right so
as you can think of this is my mining
of the normal distribution again.
As Ewing as you decrease your liability or
to sort of blurs out until it just becomes
this like blob and so as you increase your
liability your standard deviation
gets tighter and tighter so we know
that observe standard deviations are
inflated due to measurement error because
reliability is again the proportion of
observed score variance accounted for
by tree score variance and so correlations
between 2 observed variables X.
will be attenuated by measurement error in
both variables that's just a side note and
so there is a general formula for
the correction of correlations due to due
to measurement error what we do is we
divide by the square root of liabilities
and if there's if there's error and X.
and error and why we divide by the square
root of reliability in one and
the square root of reliability in the
other and this inflates the correlation
I hate this correction and I use it all
the time so because what you're sort of
trying to say is like if had these had
these variables been measured without.
That measurement error than here would
have been their correlation right
this is what structural creation models
as Matt is doing do behind the scenes for
you right there actually taking it to
actually estimating the measurement error
in each of the variables and reporting
that discipline you made a correlation for
you and and so this is a way of
sort of doing that mechanically and
in the classical test area
framework My advice here holds to
if you're going to do this report the
initial correlation and then report that
this attenuated correlation because you're
kind of doing here in a very not so
subtle way is taking advantage of
measurement error like the more
imprecision I have the greater I inflate
my test scores I mean the greater
inflating I got the greater inflate my
reliability coefficients sometimes you get
that are greater than one.
This happened and then you then you know
you've done something I mean that's just
that just reveals how silly the whole
process is right on you're giving yourself
a lot of imprecision and
credit for measurement error.
But those that that's I mean that's
something we should take away too and
then finally regression to the mean so
too much to talk about here
I'll punt this later finally so and then
finally that's going to be the this is
the correction formula that would lead
you to be suspicious of that table that I
showed you and Angela and Patrick's paper
right not suspicious in the thing and
I did something wrong but I have
questions about it right and that is that
as you increase the number of items on
your test you get greater reliability so
if you ever are in this position of
doing massive scale development and
have like 200 items do not
pat yourself on the back for
having a reliability of point 13
because you have hundreds of items
of course you do that's going to be
the average of that is going to be very
very stable with respect to measurement
error so that's why I always when I report
reliability is I also report the number of
items because you sort of condition your
interpretation of the reliability itself
on the number of items that you've got.
And so this is just an example if you
know if the liability is point one and
we double the test length what is
a predictive reliability so K.
would be 2 in this case in
the same way you can given any
given any test score length and
reliability you could estimate
the reliability of a single item test by
plugging in cases like one over the number
of items so if you ever really really want
to take a gamble that people do this right
of everything that's questions like
Would you recommend this to a friend.
That's like what's called the Net Promoter
Score and so the net promoter score is
supposed to be this like one shrew item
that tells you whether or not your product
is going to do well in the business sense
it's like a single item scale right so
anyway like if you ever want to figure
out what you know what the one item
reliability the one item test would be
just like in case one over your number
of items so these are all super handy
formulas that I would expect you to have
just kind of like in your back pocket
the way you have a standard deviation
the way you have a correlation
coefficient these are the basics.
I'm going to skip comebacks Alf see OK.
So what what is reliability what is
reliability so I said reliability
is point 8 and you're trying to explain to
your your uncle what point what you say.
And you can talk generally about
reliability is some sort of measure of
that's good but I also know what is
point 8.8 what is what does that mean
actually it's a hard question to.
Me because the good news is.
Good good good so that's that's
the right that's the you know the sort
of coherence of the overall measure and
it's you know on the sort of 0 to one
scale right but so but then if you want
to get very specific and actually address
the magnitude itself what would it what's
clear what his point is in that case
I did my usual motor mouth routine and
like I said it a couple times but
like very quickly and without pausing.
Good that's a good that's a good rule of
thumb that's segmentations cringe at rules
of thumb but that said it's one
that I don't mind cosigning for
general purposes but so all the more
reason to know what point 7 means right.
So that you're talking about a signal to
noise ratio where you're talking about
the true score variance over
the the air variance you're close and
it's just a convolution but to anyone
you've got true score variance in
the numerator that's good
what's in the nominator.
It's absurd score variances in
the nominator total variance so
how much of the variance that you see is
accounted for by that signal and you can
get to that from the signal to noise ratio
but but but reliable so when you see
point 8 you're saying 80 percent of the
observed score variance is accounted for
by true score variance that's not
the only way to think about the.
Reliability question if you can also
frame it in just the way we think of
an intra class correlation.
As as a correlation in itself.
And it's a correlation in this case of 2
replications of the measurement procedure.
It's a correlation of X.
prime that's actually why you write
it Row X.
It's a correlation between X.
and what we imagine a replication of X.
to be which is equivalent to the
proportion of the observed score variance
accounted for that use governs so it's
bilingual in the same way that you can
think of an and enter class correlation as
a correlation and this measure
of between group variance right.
In the same way reliability is both
the proportion of observed score
variance accounted for
by trees grow variance and
the correlation between 2
applications of an event procedure.
The monster that I like is.
A person with one watch
knows what time it is
a person with 2 watches is never
quite sure and that's kind of kind of
what psychometrics is all about it's very
sort of saying like we always want to know
exactly how imprecise one to
be precise about or in prison.
OK so how do we estimate this in practice
here are 3 types of reliability The 1st
is sort of the gold standard to sort of
parallel forms reliability we actually try
to do that we try to replicate the whole
measurement procedure twice we sort of
we could we create 2 different equivalent
forms imagine to spurn the satisfactions
earn this magical turn right off of stuff
like marbles or sort of take a scoop
of the items and create one form take
another random scoop of the items and
create another form and then we give it
all to you like now and we give it all to
you in like some separate room on Sun
separate day with some separate Raiders
and we try to vary all the things that
we care about varying and give that to
give that in a different scenario and then
we simply take the correlation of the X.
prime and that's a parallel
forms reliability another
way we approach it is to do test retest
reliability what that does not capture
is the variance to the items because if
I test you and then I retest you again
I haven't drawn again from this pattern of
items so you want to think about all these
turns of like items of Raiders of
occasions of tasks and think about all of
those it's contributing to your sources
of variance and 3rd this is sort of
the weakest form that you usually get the
highest reliability from is our internal
consistency reliability which which treats
all this stuff like all the stuff that's
going on in this room right now is fixed
and only considers the variance of items
within the within the little test that you
happen to have right it sort of says hey
instead of drawing an urn drawing from
this urn of items I recognize that I've
already drawn from the urn of items I can
split the test items sort of randomly in
half to correlations of all those halves
and think about how that is an estimate of
a liability that's how internal
consistency reliability works.
So again I hope you're sort of bilingual.
In the order of the light.
consistency reliability 10 percent of
the time it's some weird approach using
R.T. that I'll talk about shortly.
And actually show Shawn and
I from our from our 2015 paper to
have this here I might have cut the slide
we actually show you the histogram for
all reported state reliability causations
that you see in practice just a give you
a sense and point to point and
all of them are over point 7 in this case
there are centered on point 9
with a slight negative skew.
And then averages of that.
Here is what I hear is
what I skipped over so
Comdex Alpha is exactly that and it
actually can show you can prove that it is
the average of all possible split halves
right you split in half you split
in half every single possible way you can
you take the correlation over and over and
over again now that correlation this is
where you combine come back south and
Spearman Brown right you've taken half
tests when you split in half you've taken
have tests so you have successfully
described on average the reliability of
a half test and then used Aaron Brown to
ramp that up to that to the full test so
it's a nice and neat little exercise.
But you've got you've got
the intuition Exactly.
So so you know this is the last thing
also service and it will take I think
a 5 minute break that will end up being
this is this is where your reliability
is not a liability right and
this is the point that I think Brian was
sort of leading to is that you should
think of the reliability coefficients that
you get in your technical manuals and
all of your state tests as being
an impoverished version of the reliability
you might imagine right if it's trying to
answer the question how well does this X.
correlate with this possible X.
prime like that's not varying items
doesn't cut it right and if you
were to vary occasions if you were to vary
like spin areas if you were to vary raters
if you were to vary all these other things
that we might actually be interested in
generalizing over and that reliable you'd
probably almost assuredly be lower right
and so that's worth thinking about as you
as you are adjusting for reliability is
what exactly are the replications over
which I'm interested in generalizing and
that leads to an entire series another yet
another theory called generalize ability
theory which is developed by the Crown doc
and many others decades ago Bob Brennan
has done the biggest the most work on this
as of late and is something you should
know about that's not going to dig too far
into right now but I'll give you a couple
key references brand in 2002 is a great
primer by my former 2nd advisor rich
Ableson in the arena lab in 1901
it's a nice little Sage primer and
it's kind of depressing that it hasn't
really gone out of date since 1901 but
this is basically just analysis of
variance is pretty straightforward.
And that's just it answers
a couple of questions Tom Kane and
I did a paper on this I present in my
class about teacher observations right and
how many readers do you need how many
lessons do you need how many items do you
need to get sufficiently precise
estimates of teacher observation scores
by many readers and are for example
administrator raters different than peer
Vader's these are the kinds of question
you generalize ability theory is.
Really well primed to answer this is my
colleague Heather Hill at Harvard who
wrote a great article on education
researchers say the title is
that before the colon was in her like
Rader reliability is not enough which is
to say like Often times we think we've got
a bunch of readers let's just see how well
they match with Master coders not
enough and I totally agree with her so
I think you should sort of think as you're
developing if you ever have a skill that
depends on Raiders you should definitely
start with greater accuracy and
then move quickly to generalize
about the theory if you can and
leverage the sources here so this
generalized ability studies are expensive
but they also are due
diligence when it comes to
real reliability right they're a liability
you have is not the reliability you seek.