ScoringAlgorithm
AnthonyG.Greenwald
UniversityofWashington
UniversityofVirginia
BrianA.Nosek
MahzarinR.Banaji
HarvardUniversity
InreportingImplicitAssociationTest(IAT)results,researchershavemostoftenusedscoringconven-tionsdescribedinthefirstpublicationoftheIAT(A.G.Greenwald,D.E.McGhee,&J.L.K.Schwartz,1998).DemonstrationIATsavailableontheInternethaveproducedlargedatasetsthatwereusedinthecurrentarticletoevaluatealternativescoringprocedures.Candidatenewalgorithmswereexaminedintermsoftheir(a)correlationswithparallelself-reportmeasures,(b)resistancetoanartifactassociatedwithspeedofresponding,(c)internalconsistency,(d)sensitivitytoknowninfluencesonIATmeasures,and(e)resistancetoknownproceduralinfluences.Thebest-performingmeasureincorporatesdatafromtheIAT’spracticetrials,usesametricthatiscalibratedbyeachrespondent’slatencyvariability,andincludesalatencypenaltyforerrors.Thisnewalgorithmstronglyoutperformstheearlier(conventional)procedure.
TheImplicitAssociationTest(IAT)providesameasureofstrengthsofautomaticassociations.Thismeasureiscomputedfromperformancespeedsattwoclassificationtasksinwhichassociationstrengthsinfluenceperformance.Theapparentuseful-nessoftheIATmaybeduetoitscombinationofapparentresistancetoself-presentationartifact(Banse,Seise,&Zerbes,
AnthonyG.Greenwald,DepartmentofPsychology,UniversityofWash-ington;BrianA.Nosek,DepartmentofPsychology,UniversityofVirginia;MahzarinR.Banaji,DepartmentofPsychology,HarvardUniversity.Therevisedscoringproceduresdescribedinthisreportareherebymadefreelyavailableforuseinresearchinvestigations.SPSSsyntaxforcom-putingImplicitAssociationTestmeasuresusingtheimprovedalgorithmcanbeobtainedattheUniversityofWashingtonWebsite(http://faculty.washington.edu/agg/iat_materials.htm).However,theimprovedscoringproceduresdescribedinthisreport(patentpending)shouldnotbeusedforcommercialapplicationsnorshouldtheyorthecontentsofthisreportbedistributedforcommercialpurposeswithoutwrittenpermissionoftheauthors.
ThisresearchwassupportedbythreegrantsfromNationalInstituteofMentalHealth:MH-41328,MH-01533,andMH-57672.TheauthorsaregratefultoMaryLeeHummert,KristinLane,andDeborahS.Mellottforhelpfulcommentsonanearlierversion,andalsotoLaurieA.RudmanandEliotR.Smith,whocommentedascolleaguesratherthanasconsultingeditorsforthisjournal.
CorrespondenceconcerningthisarticleshouldbeaddressedtoAnthonyG.Greenwald,DepartmentofPsychology,UniversityofWashington,Box351525,Seattle,Washington98195-1525.E-mail:agg@u.washington.edu
2001;Egloff&Schmukle,2002;Kim&Greenwald,1998),itslackofdependenceonintrospectiveaccesstotheassociationstrengthsbeingmeasured(Greenwaldetal.,2002),anditseaseofadaptationtoassessabroadvarietyofsociallysignificantassoci-ations(seeoverviewinGreenwald&Nosek,2001).
TheIAT’smeasure,oftenreferredtoastheIATeffect,isbasedonlatenciesfortwotasksthatdifferininstructionsforusingtworesponsekeystoclassifyfourcategoriesofstimuli.Table1de-scribesthesevensteps(blocks)ofatypicalIATprocedure.
ThefirstIATpublication(Greenwald,McGhee,&Schwartz,1998)introducedascoringprocedurethathasbeenusedinthemajorityofsubsequentlypublishedstudies.Thefeaturesofthisconventionalalgorithm(seeTable4laterinthearticle)include(a)droppingthefirsttwotrialsoftesttrialblocksfortheIAT’stwoclassificationtasks(Blocks4and7inTable1),(b)recodinglatenciesoutsideoflower(300ms)andupper(3,000ms)bound-ariestothoseboundaryvalues,(c)log-transforminglatenciesbeforeaveragingthem,(d)includingerror-triallatenciesintheanalyzeddata,and(e)notusingdatafromrespondentsforwhomaveragelatenciesorerrorratesappeartobeunusuallyhighforthesamplebeinginvestigated.Themainjustificationfororiginallyusingtheseconventionalprocedureswasthat,comparedwithseveralalternativeproceduresoftenusedwithlatencydata,theconventionalprocedurestypicallyyieldedthelargeststatisticaleffectsizes.
Previoustheoreticalandmethodologicalanalyseshaveprovidedmethodsofdealingwithproblemsthatoccurinlatencymeasures
JournalofPersonalityandSocialPsychology,2003,Vol.85,No.2,197–216
Copyright2003bytheAmericanPsychologicalAssociation,Inc.0022-3514/03/$12.00DOI:10.1037/0022-3514.85.2.197
197
198
GREENWALD,NOSEK,ANDBANAJI
Table1
SequenceofTrialBlocksintheStandardElection2000(Bushvs.Gore)IAT
Block1234567
No.oftrials20202040202040
FunctionPracticePracticePracticeTestPracticePracticeTest
Itemsassignedtoleft-keyresponseGeorgeBushimagesPleasantwords
PleasantwordsϩBushitemsPleasantwordsϩBushitemsAlGoreimages
PleasantwordsϩGoreimagesPleasantwordsϩGoreimages
Itemsassignedtoright-keyresponse
AlGoreimagesUnpleasantwords
UnpleasantwordsϩGoreitemsUnpleasantwordsϩGoreitemsGeorgeBushimages
UnpleasantwordsϩBushimagesUnpleasantwordsϩBushimages
Note.Forhalfthesubjects,thepositionsofBlocks1,3,and4areswitchedwiththoseofBlocks5,6,and7,respectively.TheprocedureinBlocks3,4,6,and7istoalternatetrialsthatpresenteitherapleasantoranunpleasantwordwithtrialsthatpresentedeitheraBushorGoreimage.TheprocedureusedfortheElection2000IATreportedinthisarticledifferedfromthisstandardprocedurebyincluding40practicetrialsinBlock6.TheprocedurefortheraceIATreportedinthisarticledifferedfromthestandardprocedurebyusing40practicetrialsinBlock5.Thesestrategieswereusedsuccessfullytoreducethetypicaleffectoforderinwhichthetwocombinedtasksareperformed.IATϭImplicitAssociationTest.
intheformofspeed–accuracytradeoffs(e.g.,Wickelgren,1977;Yellott,1971),age-relatedslowing(e.g.,Faust,Balota,Spieler,&Ferraro,1999;Ratcliff,Spieler,&McKoon,2000),andspuriousresponsesthatappearasextremevalues(oroutliers;Miller,1994;Ratcliff,1993).Remarkably,researchpracticeincognitiveandsocialpsychologyhasbeennomorethanmildlyinfluencedbythismethodologicalwork.Thatlimitedinfluencemaybeexplainedbythreepracticalconsiderations:First,someofthemethodologicalrecommendationsarecostlytouse—forexample,severalhoursofdatacollectionwitheachsubjectmaybeneededtoobtaindatasetsfromwhichindividual-subjectspeed–accuracytradeofffunctionscanbeconstructed.Second,journaleditorsandreviewersrarelyinsistonthemorepainstakingmethods.Third,researcherswhousethemoresophisticated(andpainstaking)methodsarerarelyre-wardedfortheirextrawork—conclusionsbasedonthemoreeffortfulmethodsoftendivergelittlefromthosebasedonsimplermethods.
TheconventionalscoringprocedurefortheIAThasnotprevi-ouslybeensubjecttosystematicinvestigationsofpsychometricproperties.Additionally,theconventionalscoringprocedurelacksanytheoreticalrationalethatdistinguishesitfromotherscoringmethods(Greenwald,2001).Consequently,theauthorswelcomedafortuitousopportunitytocomparetheconventionalprocedurewithalternatives.ThisopportunityarosethroughtheoperationofaneducationalWebsite(http://www.yale.edu/implicit/)atwhichseveralIATprocedureshadbeenmadeavailablefordemonstrationusebydrop-invisitors.
ThisarticlefirstdescribestheIATWebsiteandthenpresentsaseriesofstudiesthatweredesignedtoevaluatecandidatealterna-tivescoringproceduresforIATsthatoperatedontheWebsite.Theinvestigatedscoringmethodsincluded(a)transformationsoflatencymeasures,(b)proceduresfordealingwithextreme(slowandfast)responses,(c)replacement(penalty)schemesforerrortrials,and(d)criteriaforidentifyingarespondent’sdataasunfitforcomputingIATmeasures.Thearticleconcludesbyrecom-mendingareplacementfortheconventionalIAT-scoringalgorithm.
GeneralMethod
TheYaleIATWebSite
TheYaleIATWebsitewasintendedtofunctionastheInternetequiv-alentofaninteractiveexhibitatasciencemuseum.ThesitewasdesignedtoallowWebvisitorstoexperiencewhattheauthorsandmanylaboratorysubjectshaveexperienced:inabilitytocontrolthemanifestationsofauto-maticassociationsthatareelicitedbytheIATmethod.Drop-invisitorscouldtakedemonstrationversionsofIATsthathadbeeninlaboratoryusefor2–4years.Within5–10min,avisitortotheWebsitecouldcompleteameasureofimplicitattitudeorstereotype,afteroptionallyrespondingtosomeitemsthatrequesteddemographicinformationandexplicit(self-report)measuresofthetargetattitudeorstereotype.1UnlikelaboratoryIATs,theWebsiteIATsprovidedrespondentswithasummaryinterpretationoftheirtestperformancebycharacterizingitasshowing“strong,”“medium,”“slight,”or“littleorno”associationofthetypemeasuredbyeachtest.Respondentscouldalsoinspectdistributionsofsummaryresultsforlargenumbersofpreviousrespondents.Amplifyingtheusualdebriefingprocedureofanexperiment,theWebsitealsopro-videdanswerstonumerousquestionsconcerningtheIAT’smethodsandinterpretations,includingadiscussionofthedistinctionbetweentheim-plicitprejudicethattheIATsometimesmeasuresandthemoreordinarymeaningof(explicit)prejudice.2Approximately1.2milliontestswerecompletedattheYaleIATWebsitebetweenOctober1998andMay2002,whenthepresentanalyseswerebegun.
TherationaleforinterpretingtheIAT’sassociationstrengthmeasuresasindicatorsofsocialcognitiveconstructssuchasimplicitattitudeorimplicitstereotyperestsontheoreticaldefinitionofthoseconstructsintermsofconcept–attributeassociations.ThistheoreticalconceptionhasbeendescribedbyGreenwald,Banaji,Rudman,Farnham,Nosek,&Mel-lott(2002).2Thisdistinctionisdescribed,onaWebpageofanswerstofrequentlyaskedquestionsforanIATdesignedtomeasureimplicitraceattitudes,asfollows:“Socialpsychologistsusetheword‘prejudiced’todescribepeoplewhoendorseorapproveofnegativeattitudesanddiscriminatorybehaviortowardvariousout-groups.ManypeoplewhoshowautomaticWhitepreferenceontheBlack–WhiteIATarenotprejudicedbythisdefinition.
1IMPROVEDSCORINGOFTHEIAT
199
Respondents
Recruitment.Recruitmentoccurredviamediacoverage,linksfromothersites,linksprovidedbysearchengines,andwordofmouth.Mediacoveragemayhavebeenthemostsignificantinfluenceonresponserate.Forexample,over150,000visitstotheYaleIATsitewererecordedinthe5daysfollowingtelevisedprogramsthatdescribedtheIATontheNationalBroadcastingCompany(NBC)televisionprogram,Dateline(March19,2000)andonaDiscoveryChannelprogramtitledHowBiasedAreYou?(March20,2000).Thedataanalyzedinthisreportwereprovidedbyrespondentsina9-monthperiodbetweenJuly2000andMarch2001.Characteristicsofrespondents.TheIATWebsiteincludedapromi-nentassurancethatanonymityofvisitorswouldbeprotected.Becauseofthisanonymity,theWebsitedataprovidednoopportunitytotrackcharacter-isticsofrespondentsbeyondtheiroptionalresponsestosomeself-reportquestionsthatappearedonthesite.Approximately90%ofrespondentsdid,however,respondtosomeorallofthedemographicquestions.Oftheserespondents,61%werefemaleand39%weremale;60%werebelow24yearsofage,36%werebetween24and50,and4.6%wereover50;0.7%wereNativeAmerican,6.4%wereAsian,5.0%wereBlack,3.8%wereHispanic,76.0%wereWhite,1.0%werebiracial(Black–White),3.3%weremultiracial,and4.0%reported“other”forethnicity;18%reportedhavingahighschooldiplomaorlesseducation,47%hadsomecollegeexperience,21%hadabachelor’sdegree,and14%hadapostbaccalaureatedegree;80%oftherespondentsreportedbeingfromtheUnitedStatesand,ofthe20%non-U.S.respondents,abouthalfcamefromCanada,Australia,orBritain(evenlydistributed),andtheremainderfromothercountries.
Procedure
Materialsandapparatus.WebsiteIATswerepresentedusingJavaAppletandCommonGatewayInterface(CGI)technology.Afteritwasdownloadedviatherespondent’sbrowser,theprogramusedtherespon-dent’scomputertopresentstimuliandtomeasureresponselatencies.Therespondent’sbrowserprogramreturnedtherespondent’sdatatotheWebserver.Theserverthenanalyzedthedataandreportedatestresultwithinseveralseconds.Testresultswerereportedasshowing“strong,”“medi-um,”“slight,”or“littleorno”strengthofoneoftheassociationcontrastsmeasuredbythetest.3Forexample,fortheRaceIAT,theresultsindicatedthestrengthofrespondents’automaticpreferencesforBlackrelativetoWhiterace—thatis,differentialassociationofBlackandWhitewithpleasant.Precisioninmeasuringindividuallatencieswaslimitedbytheclockrateoftheoperatingsystemthatsupportedtherespondent’sWebbrowser(e.g.,18.2HzforWindowssystems).Thiswasnotadebilitatinglimitationbecauseofthenonsystematicnatureoftheresultingnoiseandthesubstantialreductionofitsmagnitudeproducedbyaveragingdataoverapproximately40trials.
Self-reportmeasuresanddemographicdata.BeforeeachIAT,respon-dentsreceivedanoptionalsurveypagethatincludeditemstomeasureexplicitattitudesorbeliefsregardingtheIAT’stargetcategoriesalongwithsomedemographicitems.Respondentswereinformedthattheself-reportanddemographicitemswereoptional—respondentscouldproceedtotheIATdemonstrationwithoutrespondingtotheitems.
IATmeasures.NineIATmeasureswereavailableattheYaleIATsiteatvarioustimesstartinginlateSeptember1998:implicitraceattitude,usingeither(a)AfricanAmericanandEuropeanAmericanfirstnamesor(b)mor-phedraciallyclassifiablefacesandtheattributesofgoodandbad;implicitageattitude,usingeither(c)firstnamesor(d)morphedage-classifiablefacesand
Thesepeopleareapparentlyabletofunctioninnon-prejudicedfashionpartlybymakingactiveeffortstopreventtheirautomaticWhitepreferencefromproducingdiscriminatorybehavior”(https://implicit.harvard.edu/implicit/demo/racefaqs.html).
thegood–badattributecontrast;(e)implicitgender–careerstereotype,mea-suringassociationoffemaleandmalewithcareerandfamily;(f)implicitgender–sciencestereotype,measuringassociationoffemaleandmalewithscienceandliberalarts;(g)implicitself-esteem,measuringassociationsofselfandotherwithgoodandbad;(h)implicitmath–artsattitude,measuringassociationsofmathandartswithgoodandbad;and(i)Election2000implicitcandidatepreference,measuringassociationscontrastingpairsofmajorcan-didatesintheU.S.presidentialprimariesof2000withgoodandbad.MoredetaileddescriptionsoftheseIATsareavailableinNosek,Banaji,andGreen-wald(2002a).FourofthenineIATs(b,d,f,andiintheprecedinglist)providedthedataforthepresentanalyses.
Sequenceoftasks.RespondentsfirstsawpreliminaryinformationthatdescribedwhattheymightexperienceintakinganIAT.Theywerethenofferedtheopportunitytocontinueiftheywishedtodoso.ThosewhocontinuedthenchoseoneIATfromalistoffourtosixthatwerecurrentlyavailableontheWebsite.Third,respondentsoptionallyreportedtheirattitudesorbeliefsinresponsetooneormoreself-reportitemsthatwerewordedtocapturethecomparisonofconcepts(e.g.,preferenceforyoungvs.old)usedintheupcomingIATmeasure.Fourth,respondentsoptionallyrespondedtodemographicitems.Fifth,respondentsreadinstructionsfortheWeb-administeredIATandproceededtocompleteit.CompletionofanIATtypicallyrequired5–10min.Preliminaryinformationadvisedrespon-dents(a)aboutpossiblediscomfortsthatmightbeproducedbythetest’sspeedstressanditsuseofvisualstimuli,(b)thatthereportedresultsofthetestwerenotguaranteedtobevalid,and(c)thattherewasnoobligationtocompletetheIATafterstartingit.
LimitationsoftheWebSiteData
Self-selection.Therespondentsamplesforthisresearchcannotbetreatedasrepresentativeofanydefinablepopulation.Atthesametime,thesamplewasconsiderablymorediversethantypicalresearchsamples(seeCharacter-isticsofrespondents).Animportantfeatureofthesampleswastheirlargesize,whichaffordedthestatisticalpowertodiscriminatesmall,butpossiblycon-sequential,differencesinpropertiesofalternativescoringprocedures.
Possiblemultipleparticipationsbyrespondents.BecauseparticipationattheIATWebsitewasanonymous,WebsitevisitorscouldcompleteasmanyIATsastheywishedandcouldtakethesameIATmultipletimes.Multipledatapointsfromsinglerespondentsposeobviousproblemsforstatisticalanalysis.However,theoveralllargenumberofrespondentsreducesthepotentialimpactofthisproblem:Few,ifany,singlerespon-dentscouldplausiblyhaveprovidedasmuchas0.1%(e.g.,10in10,000observations)ofanyofthedatasets.ForadditionaldiscussionofmultipledatapointsfromsinglerespondentsseeNoseketal.(2002a).OneofthepreliminaryoptionalquestionsgiventorespondentsaskedhowmanyIATstheyhadpreviouslycompleted.Thatmeasurewasavailabletoassesstheeffectofpriorparticipation.
CriteriaforEvaluatingCandidateIATMeasures
EachofthefollowingcriteriaforevaluatingIATmeasureswasusedinoneormoreofthepresentseriesofstudies.Thefirsttwocriteria,IATcorrelationswithexplicitmeasures(highcorrelationsdesired)andcorre-lationwithaveragelatency(lowcorrelationsdesired)arethemostimpor-tantonesofthefollowingsixcriteria.
IATcorrelationswithexplicitmeasures.Threeself-reportitemswereavailableforcomparisonwitheachIAT.OnewasaLikert-typemeasurethatrequestedacomparativeappraisalofthetwoopposedtargetconcepts(e.g.,youngvs.oldfortheAgeIAT)ontheIAT’sattributedimension
3Theslight,medium,andstronglabelscorrespondedtoresultsmeetingtheconventionalcriteriaforsmall,medium,andlargeeffectsizesofCohen’s(1977)dmeasure.
200
GREENWALD,NOSEK,ANDBANAJI
(positivevs.negativevalenceforthreeoftheIATs;sciencevs.artsforthefourth).Thesecondandthirdself-reportitemswereinthermometerformat,requestingseparatejudgmentsfortheIAT’stwotargetconceptsonan11-pointscalefortheIAT’sattributedimension.(Thethermometerscaleshadjust5pointsfortheGender–ScienceIAT.SeetheAppendixforwordingsofallexplicit-measureitems.)
Bysubtraction,thetwothermometeritemswerecombinedintoather-mometerdifferencescore.TheLikertmeasureandthethermometerdif-ferencemeasureswerethencombinedintoanoverallexplicitmeasurebystandardizingeachandaveragingthetworesultingscores.
CorrelationsoftheoverallexplicitmeasurewiththevariousIATmea-sureswerecomputed.Althoughvaluesforimplicit–explicitcorrelationsvariedwidelyforthefourdatasets,allwerepositive,consistentwithpreviousobservations(Noseketal.,2002a).Usingtheconventionalalgo-rithmforscoringtheIAT,implicit–explicitcorrelationswere.11,.20,.29,and.69,respectively,fortheAge,Gender–Science,Race,andElection2000IATs.VariationsamongthesecorrelationsareassumedtoresultfromvariationsintheextenttowhichIATandself-reportshareinmeasuringtheassociationsthattheIATisintendedtomeasure(e.g.,fortheAgeIAT,associationsofyoungoroldwithpleasantorunpleasant).
Acentralassumptionforanalysesinthisarticleisthathigherimplicit–explicitcorrelationsforamodifiedIATmeasurecanindicategreaterconstructvalidityofthemodifiedmeasureasameasureofassociationstrengths.Thiscentralassumptiondependsonafurtherassumptionthatassociationstrengthisalatentcomponentofboththeimplicitandexplicitmeasures.Theimportanceoftheshared–latentcomponentassumptioncanbeillustratedbyanalogytothewayinwhichasuperiormeasureofheightshouldincreasethecorrelationbetweenheightandweight.Inthecaseofheightandweight,thesharedlatentcomponentisheight,inthesensethatweightcanbeunderstoodashavingcontributionsduetoheight,girth,anddensity.Inthiscircumstance,animprovedheightmeasure(e.g.,arulerthatcanbereadtothenearesthalfinchratherthantothenearesthalffoot)shouldyieldhighercorrelationswithweight.
Justasforimplicit–explicitcorrelations,thecorrelationbetweenheightandweightcanvaryconsiderablyfordifferentsamples.Forexample,heightandweightmaybecorrelatedalmostperfectlywhenotherdetermi-nantsofweight(girthanddensity)areeitherkeptconstantorarecorrelatedwithheight,asmightbethecaseforasampleofnewborninfants.Bycontrast,forasampleofAmericanprofessionalfootballplayers,theheight–weightcorrelationmaybemuchlowerbecauseheightsmayvarylittleandgirthsmayvaryconsiderably.Nevertheless,ineithersample(newbornsorfootballplayers),theheight–weightcorrelationshouldbelargerforamoresensitivemeasureofheight.Theinterpretationofimplicit–explicitcorrelationsasindicatorsofconstructvalidityofIATmeasuresisconsideredfurtherintheGeneralDiscussion.
CorrelationsofIATwithresponselatency.Researchoncognitiveaginghasestablishedthateffectsofexperimentaltreatmentsonresponselatencyaregenerallylargerforelderlythanforyoungsubjects.Thisagedifferenceisknowntobeassociatedwithgreateraveragelatencyforelderlysubjects(age-relatedslowing;e.g.,Brinley,1965;Faustetal.,1999;Ratcliffetal.,2000).Consequently,itisexpectedthatIATeffectswillbeartifactuallylargerforanysubjectswhorespondslowly,notjusttheelderly.ThisartifactshouldtaketheformofapositivecorrelationofextremityofIATeffectswithresponselatency.4ItisdesirableforanIATmeasuretominimizethisundesiredartifactualcorrelationwithresponsespeed.
Internalconsistency.Foreachcandidatescoringalgorithm,twopart-measureswerecreatedbyapplyingthescoringalgorithmseparatelytotwomutuallyexclusivesubsetsoftheIAT’scombined-tasktrials.Thecorre-lationbetweenthesetwopart-measures,acrossrespondents,providedameasureofinternalconsistency.
Sensitivitytoknowninfluences.ThreeoftheIATsincludedinthisresearchwereknowntobesensitivetoimplicitattitudesandstereotypesthatarepervasivein(atleast)Americansociety.TheAgeIATtypically
indicatesstrongimplicitpreference5foryoungrelativetoold,andtheGender–ScienceIATtypicallyindicatesstrongmale–scienceandfemale–artsassociations.FortheRace(Black–White)IAT,thetypicalpatternisimplicitpreferenceforWhiterelativetoBlack.Sensitivitytotheseknownmodalresponsetendencieswasusedasanindicatorofperformanceforthealternativescoringalgorithms.Useofthiscriterionisbasedontheassump-tionthatthemodalresponsetendenciesreflectpopulationdifferencesinassociationstrengths.Thatassumptionisconsistentwithmuchresearchevidence(e.g.,Asendorpf,Banse,&Mu¨cke,2002;Ashburn-Nardo,Voils,&Monteith,2001;Egloff&Schmukle,2002;Gawronski,2002;Green-waldetal.,2002;Greenwald&Nosek,2001;McConnell&Leibold,2001;Nosek,Banaji,&Greenwald,2002b;Rudman,Feinberg,&Fairchild,2002),althoughsomealternativeinterpretationshavebeensuggested(e.g.,Brendl,Markman,&Messner,2001;Rothermund&Wentura,2001).Resistancetoundesiredinfluenceoforderofcombinedtasks.AnalysesofWebsiteIATdatabyNoseketal.(2002a)confirmed,inWebsiteIATs,afindingoriginallyreportedbyGreenwaldetal.(1998):IATmeasurestendtoindicatethatassociationshavegreaterstrengthwhentheyaretestedinthefirstcombinedtask(seeTable1,Blocks3and4)thaninthesecondcombinedtask(Blocks6and7).Ontheassumptionthatassociationstrengthsarenotalteredbytheorderofcombinedtasks,anIATmeasurethatminimizesthisproceduraleffectisdesirable.
ResistancetoeffectofpriorexperiencetakinganIAT.AnalysisofWebsiteIATsbyNoseketal.(2002a)indicatedthatIATmeasuresarereducedinextremityforrespondentswhohavepriorexperiencetakingoneormoreIATs.OntheassumptionthattakingtheIATdoesnotaltertheassociationstrengthsbeingmeasured,anIATmeasurethatminimizesthisproceduraleffectisdesirable.
CandidateMeasures
TheIATmeasurehasconventionallybeencomputedasthedifferencebetweencentraltendencymeasuresobtainedfromitstwotestblocks,whichareBlocks4and7inTable1.Thepresentresearchstartedbyselectingfivecandidatemethodsofcomputingthisdifference.
Median.Themedianofeachtestblockwasusedastheblock’ssummarymeasure.ThedifferencebetweenthetwomediansprovidedtheIATmeasure.Themedianisusedrelativelyinfrequentlywithlatencydependentmeasures.Itwasincludedheremainlybecauseofcuriosityaboutitsperformanceincomparisonwithothermeasures.
Mean.Thearithmeticmeanlatencywascomputedforeachtestblock.TheresultingIATmeasurewasthedifferencebetweenthetwomeans.ThismeasureistypicallyusedforgraphicortabularpresentationofresultsinIATresearch,buthasbeeninferiortotheconventional(log)measureinstatisticaltests.
Log.Themeasureforeachtestblockwasthemeanofnaturalloga-rithmtransformationsofindividual-triallatencies.TheIATmeasurewasthedifferencebetweenthesemeans.ThisisthetransformationthathasbeenconventionallyusedinstatisticaltestsofIATmeasures(e.g.,analysesofvariance,correlations,regressions,andeffectsizecomputations).Therationaleforthelogtransformationisprovidedbythetypicallyextendeduppertailsoflatencydistributions.Thelogtransformationimprovesthesymmetryoflatencydistributionsbyshrinkingtheuppertailandistherebyexpectedtoimprovecentraltendencyestimates.
Reciprocal.Themeasureforeachblockisthemeanofreciprocallatencies(computedas1,000Ϭlatency).TheIATmeasureisthediffer-
4Thisarticleprovidesclearevidencefortheexistenceofthisartifact(seeFigure2).5TheIATmeasuresrelativestrengthsofassociations.“Implicitprefer-ence”isashorthandforstrongerassociationofoneofthetwotargetconceptswithpositivevalence,and/orweakerassociationofthatconceptwithnegativevalence.
IMPROVEDSCORINGOFTHEIAT
201
encebetweenthesemeans.Likethelogtransform,thereciprocalimprovesthesymmetryofdistributionswithextendeduppertails.Tokeepdirec-tionalityofmeasuresthesameforallIATmeasures,thedifferencescoreforthereciprocalmeasurewasreversedbysubtractingitfromzero.
D.Thismeasuredividesthedifferencebetweentestblockmeansbythestandarddeviationofallthelatenciesinthetwotestblocks.Partoftherationaleforthisadjustmentisthatmagnitudesofdifferencesbetweenexperimentaltreatmentmeansareoftencorrelatedwithvariabilityofthedatafromwhichthemeansarecomputed.Usingthestandarddeviationasadivisoradjustsdifferencesbetweenmeansforthiseffectofunderlyingvariability.Arelatedadjustmenthasbeenrecommendedforuseincogni-tiveagingstudies,inwhichtreatmenteffectsonlatenciesareoftengreaterforelderlysubjects,whoshowbothhighermeansandgreatervariabilityoflatenciesthanyoungsubjects.(Fordiscussionsofthevariabilityproblemincognitiveagingstudiessee,e.g.,Brinley,1965;Faustetal.,1999;Ratcliffetal.,2000).Asuccessfulexploratoryattempttousethistypeofindividual-variabilitycalibratedmeasurewasrecentlyreportedbyHum-mert,Garstka,O’Brien,Greenwald,andMellott(2002).
Divisionofadifferencebetweenmeansbyastandarddeviationisquitesimilartothewell-knowneffect-sizemeasure,d(Cohen,1977).ThedifferencebetweenthepresentDmeasureandthedmeasureofeffectsizeisthatthestandarddeviationinthedenominatorofDiscomputedfromthescoresinbothconditions,ignoringtheconditionmembershipofeachscore.Bycontrast,thestandarddeviationusedincomputingtheeffectsizedisapooledwithin-treatmentstandarddeviation.Toacknowledgeboththismeasure’ssimilaritytodanditsdifference,thepresentmeasureisidentifiedwithanitalicizeduppercaseletter(D)ratherthananitalicizedlowercaseletter.6AnalysisandReportingStrategy
Thepresentseriesofstudiesexaminedalternativepoliciesforretainingtrials,includingpracticetrialsanderrortrials,inthedataset(Study1);alternativedatatransformations(Study2);useofcriteriabasedonspeedoraccuracyofrespondingasthebasisfordiscardingrespondentsfromthedataset(Study3);applyingtimepenaltiesfortheoccurrenceoferrors(Study4);anddeletingextreme(fastorslow)latenciesorrecodingthemtoupperandlowerboundaryvalues(Study5).
Tokeepthetaskofexploringalternativescoringproceduresmanage-able,Studies1–5focusedonthetwomostimportantperformancecriteria:magnitudeofimplicit–explicitcorrelationofIATscoreswithself-reportandresistancetocovariationoftheIATmeasurewithlatencydifferencesamongrespondents.Study6examinedcombinationsofthebest-performingproceduresidentifiedinStudies1–5andusedthefullsetofperformancecriteriathatwereavailabletocomparealternativescoringalgorithms.
Theseriesofsixstudies,conductedinparallelforfourlargedatasets,generatedmanymoreanalysesthancanbedescribedinthisarticle.ForStudies1–5,resultsarepresentedinsomedetailforthedatasetthathadlargestvaluesofimplicit–explicitcorrelations(Election2000candidatepreference).7ResultsofStudies1–5fortheotherthreedatasets(Age,Race,andGender–Science)arementionedinpassingwhentheyshedadditionallight.ResultsfromallfourdatasetsarepresentedforStudy6.
Study1:UsefulnessofPracticeTrialsandErrorTrials
TheconventionalIATalgorithmdiscardsthefirsttwotrialsofeachtestblock(Blocks4and7inTable1)becauseoftheirtypicallylengthenedlatencies.Additionally,theconventionalal-gorithmtreatsaspractice(andexcludesfrommeasurecomputa-tions)thetwocombined-taskblocksthatprecedethetwotestblocks(Blocks3and6inTable1).Theconventionalalgorithmalsodiffersfrommanyotheranalysesoflatencydatabyretaininglatenciesfromtrialsonwhicherrorsoccurred.Study1examined
theseexclusionsandinclusionstodeterminewhethertheycouldbejustifiedintermsoftheirimpactonperformanceoftheIATmeasure.
Method
Dataset.Allfourdatasetswereanalyzed.However,onlytheresultsfortheElection2000datasetaredescribedhereindetail.Respondentscouldchooseanytwooftheactivelycompetingcandidatesforthenomi-nationsoftheRepublicanandDemocratparties.(ThemostprominentcandidateswereGeorgeW.Bush,AlGore,JohnMcCain,andBillBrad-ley.)Analyseswerelimitedtothepairthatwasmostoftenselected,GeorgeW.BushandAlGore.
Respondents.TheU.S.PresidentialElectiontookplaceonNovem-ber7,2000.TheanalyzeddatawereobtainedbetweenOctober3,2000andMarch20,2001.Of11,956whochosetocontrastBushandGoreintheIAT,slightlyoveraquarter(26.7%)tooktheIATonorbeforeElectionDay.Another31.1%tooktheIATonorbeforeDecember13,thedayonwhichtheelectionofficiallyconcludedwiththevictoryofGeorgeW.Bush.CompleteIATdatawereavailablefor8,891respondents(3,065didnotcompletetheIAT).Ofthese,completeself-reportdata(oneLikertitemandtwothermometeritems)wereavailablefor8,218(92.4%ofthosewhocompletedtheIAT).
Preliminaryexclusionsofverylonglatencies.Thedatasetcontainedoccasionalextremelylonglatencies—someinexcessof106ms,whichismorethanaquarterofanhour.TheseextravagantlatenciescouldhavebeenproducedwhenrespondentstemporarilyabandonedtheIATinfavorofsomeotheractivity.Suchextremevaluesarenotgenerallytoleratedinanalysesoflatencydata.Hadtheybeenretainedinthepresentdatasets,theywouldhaveimpairedsomeofthecandidatemeasuresmuchmorethanothers.Atthesametime,itseemeddesirabletokeepinitialcleansingtoaminimum.Somewhatarbitrarily,then,latenciesabove10,000mswereexcludedbeforeanyfurthercomputations.
IATmeasurecomputations.Eachofthefivemeasures(median,mean,log,reciprocal,andD)involvedcomputing,first,acentraltendencymeasureforeachofthetwocombinedtasksand,second,adifferencebetweenthesecentraltendencymeasures.AllIATmeasureswerecom-putedsuchthathighernumbersindicatedimplicitpreferenceforGeorgeW.BushrelativetoAlGore.ThedifferentmeasureswerecomparedintermsofcorrelationsofIATmeasuresbothwithself-report(i.e.,explicit)measuresandwithrespondentaveragelatencies.Respondentswereclas-sifiedasself-reportedBushorGoresupportersonthebasisoftheirresponsestothe5-pointLikertitemthatassessedrelativepreferenceforBushandGore.Beforecomputingcorrelationswithaveragelatency,IATmeasuresforself-reportedGoresupporterswerereversed(subtractedfromzero)sothattheexpectedcorrelationofIATscoreswithaveragelatencieswouldbepositive.Thecorrelationswithaveragelatencywerecomputedusingthedataonlyforrespondentswhoseself-describedsupportforeithercandidatewasstrong.Thesamplecontained5,202self-characterizedstrongsupporters,ofwhom3,373(64.8%)favoredGore.
6TheauthorsconductednumerousanalysestocomparetheDanddtransformationsasIATeffectmeasures.TheDtransformationwasob-servedconsistentlytobesuperiorand,accordingly,onlyresultsforDarepresentedinthisreport.7Partofthereasonforfocusingonthisdatasetisasausefulcontrasttothelowimplicit–explicitcorrelationsthathavebeenreportedinmostpreviouspublicationsconcerningtheIAT.Althoughsuchlowcorrelationsaretypicalforattitudesandstereotypesinvolvingstigmatizedgroups,thereareimportantdomainsforwhichcorrelationsarehigher—notonlyatti-tudestowardpoliticalcandidates,butalsoattitudestowardacademicsub-jects(Noseketal.,2002b)andconsumerattitudes(Maison,Greenwald,&Bruin,2001).
202
GREENWALD,NOSEK,ANDBANAJI
ResultsandDiscussion
Firsttwotrialsofcombined-taskblocks.Thefirstanalysisexaminedeffectsoftheconventionalalgorithm’spreliminarydis-cardofthefirsttwotrialsofcombined-taskblocks(Blocks4and7inTable1).Thispracticewasoriginallybasedontheobservationthatthefirsttwotrials’latencieswere,onaverage,substantiallyslowerthantheremainderoftrialsinthesameblocks.However,theslownessoftheselatenciesdoesnotnecessarilymeanthattheirinclusionwillcontaminatemeasures.Todeterminetheusefulnessofdatafromthefirsttwotrials,twodatasetswerepreparedthatdifferedininclusionversusexclusionofthefirsttwotrialsofcombined-taskblocks.
Correlationswithself-reportmeasureswereslightlyhigherforthedatasetthatretainedthefirsttwotrials.Inaddition,correla-tionsofIATextremitywithrespondents’averagelatenciesoncombined-taskblockswereslightlylowerwithinclusionofthefirsttwotrials.Bothoftheseresultsindicatedthatthefirsttwotrialsofcombined-taskblockswereuseful,despitetheirrelativelyhighlatencies.ThispatternoccurredsimilarlyinthedatasetsfortheRace,Age,andGender–ScienceIATs.Accordingly,allofthefollowinganalysesincludedthedatafromthefirsttwotrialsofcombined-taskblocks.
DatafromBlocks3and6.Theconventionalalgorithmex-cludestrialsfromBlocks3and6,treatingthemaspractice.Toassesstheusefulnessofthesedata,separateIATmeasureswerecomputedfromBlocks3and6(practice)andfromBlocks4and7(test).Remarkably,forallfivepairsofIATmeasures(median,mean,log,reciprocal,andD),correlationswithexplicitmeasureswerehigherforthemeasurebasedonBlocks3and6thanforthemeasurebasedonBlocks4and7.Further,thedifferencewasmorethantrivial.Thelargestdifferencewasforthereciprocalmeasure(practicerϭ.635;testrϭ.478).ThisdiscoverythatpracticeblocksprovidedagoodIATmeasurewasconfirmedinthedatasetsfortheRace,Age,andGender–ScienceIATs.
Tomakeuseofthedatafrompracticeblocks,newIATmea-sureswerecomputedasequal-weightaveragesofpracticeandtestblockmeasuresforallfivetransformations.Withtheexceptionofthereciprocalmeasure,thesepracticeϩtestmeasuresyieldedhighercorrelationswithself-reportthandideitherthepracticemeasureorthetestmeasurealone.Forexample,fortheDmeasure,practicerϭ.748,testrϭ.700,andpracticeϩtestrϭ.773.CorrelationsofIATmeasureswithrespondentaveragelatencytendedtobehigherforthepracticemeasurethanforthetestmeasure.Forpracticeϩtestmeasures,thecorrelationswithaver-agelatencytendedtobesimilartothoseforpracticealone.AgainusingDfortheillustration,practicerϭ.073,testrϭ.048,andpracticeϩtestrϭ.070.
Errorlatencies.Itiscommonpracticeinstudieswithlatencymeasurestoanalyzelatenciesonlyforcorrectresponses.Bycon-trast,theconventionalIATalgorithmuseserrorlatenciestogetherwiththoseforcorrectresponses.Study1includedanalysestocomparethevalueofincludingversusexcludingerrorlatencies.ApreliminaryanalysisoftheElection2000IATdatawaslimitedtorespondents(nϭ1,904)whohadatleasttwoerrorsineachofBlocks3,4,6,and7.Theanalysisindicatedthaterrorlatencies(Mϭ1,292ms;SDϭ343)wereabout500msslowerthancorrectresponselatencies(Mϭ790ms;SDϭ301).TheincreasedlatencyoferrortrialsisexplainedbytheWebIAT’s
proceduralrequirementthatrespondentsgiveacorrectresponseoneachtrial.(ErrorfeedbackintheformofaredletterXindicatedthattheinitialresponsewasincorrect.Respondents’instructionsweretogivethecorrectresponseassoonaspossibleafterseeingtheredX.)Latenciesonerrortrialsthereforealwaysincludedtheaddedtimerequiredforsubjectstomakeasecondresponse.Asecondpreliminaryanalysis,whichwaslimitedtorespon-dentswhohadself-characterizedstrongpreferenceforeitherGoreorBush,showedthaterrorrateswerehigherwhenrespondentswererequiredtogivethesameresponsetotheirpreferredcandi-dateandunpleasantwords(Mϭ12.4%)thanwhengivingthesameresponsetotheirpreferredcandidateandpleasantwords(Mϭ5.5%).
Together,thesetwopreliminaryanalysessuggestedthatinclu-sionoferrorlatenciesshouldenhanceIATeffects.Thisenhance-mentshouldoccurbecauseerrorswereboth(a)slowerthancorrectresponsesand(b)morefrequentwhenthetaskrequiredgivingthesameresponsetononassociatedtarget–attributepairs(e.g.,thepreferredcandidateandunpleasant-meaningwords).InatestforcorrelationofIATmeasureswiththecombinedself-reportmea-sure,theDmeasureperformedbetter(rϭ.753)whenerrorlatencieswereincludedthanwhentheywereexcluded(rϭ.730).Atthesametime,thecorrelationwithaverageresponselatencywasonlyveryslightlygreater(whichisundesirable)whenerrorlatencieswereincluded(rϭ.070)thanwhentheywereexcluded(rϭ.063).Theincreaseincorrelationwithself-reportamountstoa3.0%increaseinvarianceexplainedcomparedwithanincreaseinvarianceexplainedofonly0.1%inthecorrelationofIATwithaveragelatency.Forthisreason,itappearedveryreasonabletoretainerrorlatenciesintheIATmeasures.FurtheralternativesfortreatingdatafromerrortrialsareconsideredinStudy4.
Inseveralways,Study1demonstratedthatinclusionofdataisagenerallygoodpolicyfortheIAT.Improvementsinperformancewereapparentindatasetsthatretained(a)thefirsttwotrialsofcombined-taskblocks,(b)errorlatencies,and(c)datapreviouslytreatedaspractice(Blocks3and6intheIATschemaofTable1).ThegreatestoftheseimprovementsofperformanceresultedfromincludingdatafromBlocks3and6inadditiontothosefromBlocks4and7.
Study2:ComparingFiveTransformationsofLatenciesMethod
ResultsofStudy1wereappliedinconstructingdatasetsusedforalloftheremainingstudies.ThedatasetsforStudies2–6thereforeusedalltrialsfromBlocks3,4,6,and7,includingtrialsonwhicherrorsoccurred.Withthisinclusivedataset,thefivemeasuresdescribedaboveunderCandidateMeasureswereevaluatedintermsoftheircorrelationwithexplicitmea-suresandtheirresistancetocontaminationbylatencyvariationsamongrespondents.Thesetwoperformancecriteriacouldbeevaluatedbyexam-ininglatencyoperatingcharacteristic(LOC)functions,whichareplotsofmeasuresasafunctionofthelatenciesoftheresponsesonwhichtheyarebased(e.g.,Lappin&Disch,1972).
ResultsofStudy2areshowninFigures1and2intheformofLOCplotsfortheimplicit–explicitcorrelationandforthemeanvalueoftheIATmeasure.TheexplicitmeasureusedinthecorrelationsforFigure1was(asdescribedabove)theaverage,foreachrespondent,ofstandardizedvaluesofaLikert-typemeasureofcandidatepreferenceandadifferencemeasurecreatedfromthermometer-typemeasuresoflikingforeachcandidate(BushandGore).AsapreliminarytoconstructinganyLOCplots,an
IMPROVEDSCORINGOFTHEIAT
203
Figure1.Latencyoperatingcharacteristics(LOCs)forcorrelationswithself-reportforfiveImplicitAssociationTest(IAT)scoringalgorithms.HighercorrelationsandflatterLOCcurvesindicatebetterperformance.Datapointsarecorrelationsfor20groupsofrespondents,sortedbytheirresponsespeed.DataarefromStudy2,Election2000IATdataset.Foreachcorrelation,nrangesbetween396and420.
averagelatencymeasurewascomputedforeachrespondentasanequal-weightaverageofmeanlatenciescomputedfromeachofthefourdatablocks(involvingatotalof140trials).Inthesampleof8,891respondentsforwhomthismeasurewasavailable,averagelatencieshadameanof929ms(SDϭ776)andrangedfrom215msto69,814ms.(Suchahighvaluewaspossiblebecausetheseaverageswerecomputedbeforedeletinglaten-ciesgreaterthan10,000msfromthedataset.)Usingthismeasure,20-tilesofthedistributionwereidentified.Thefirst20-tileconsistedofthe5%ofthesamplewithfastestaveragelatencies,andthelastconsistedofthe5%withslowestaveragelatencies.
ResultsandDiscussion
Figure1displayscorrelationLOCsforthemedian,mean,log,reciprocal,andDmeasures.TheseLOCsindicatebetterperfor-manceoftheIATmeasuretotheextentthattheyare(a)highinelevation(highercorrelationsindicatebetterperformance)and(b)level(i.e.,flat),indicatingconsistencyofthecorrelationacrossthewiderangeofrespondentspeeds.Onbothofthesecriteria,theDmeasureperformedbestofthefiveinvestigatedtransformations,andthereciprocalmeasureperformedworst.Thatis,theLOCfortheDmeasurewasbothhigherandmorelevelthantheLOCsfortheotherfourmeasures(seeFigure1).Differencesamongthemeasuresaremostnoticeableatthefast(left)endoftheLOCs.Themeasureusingthemeanwasthesecond-bestperformeronbothofthetwodesirablecharacteristicsandisquiteclosetothebest-performingDmeasureintheslower(right)halfoftheLOC.Figure2displaysLOCsforthemeansofthefivemeasures,usingdataforthe5,202respondentswhoindicatedstrongprefer-enceforeitherGoreorBushontheLikertself-reportmeasure.Forthisanalysis,IATvaluesforGoresupportersweresubtractedfrom
zerosothatallmeanvalueswereexpectedtobepositive.ForFigure2’sLOC,elevationisnotacriticalindicatorbecausetheseveralmeasuresusedfourdifferentnumericscalesthatarenotdirectlycomparable.(Onlythemedianandmeanshareametric.)Onthebasisofassumingthatextremityofimplicitcandidatepreferencesofslowrespondersshouldnotdifferonaveragefromthatoffastresponders,levelnessoftheLOCfunctionsinFigure2isverydesirable.FortheLOCsshowninFigure2,themeanandmedianmeasuresperformedquitepoorly.Forthemedian,thedatasuggestedthatimplicitfavorablenesstowardthepreferredcandi-dateoftheslowestresponderswasoverseventimesthatofthefastestresponders(ratioϭ7.09:1).Forthemean,thecorrespond-ingfigurewasanalmostequallypoor5.96:1.Forthelog,D,andreciprocalmeasures,thecorrespondingvalueswere,respec-tively,2.82:1,1.42:1,and1.26:1.Thus,allofthemeasurespro-ducedlargervaluesofIATmeasuresforslowthanfastresponders,butthemeasuresvariedconsiderablyintheextenttowhichtheirvalueswerecorrelatedwith(i.e.,contaminatedby)responsespeed.AsimplesummaryofFigure2’sdataisprovidedbythecorre-lationofeachIATmeasurewithresponsespeedfortheentiresubsampleofstrongsupporters.Thesecorrelationsrangedfromalowvalueofrϭ.050forthereciprocalmeasuretoahighofrϭ.344forthemean.Theothervalueswere:D(rϭ.070),log(rϭ.226),andmedian(rϭ.309).
ThebriefsummaryofStudy2isthatoverall,theDmeasureperformedbest.Itshowedclearlythebestperformanceonthecriterionofimplicit–explicitcorrelationandwassecondbestinNOTE: The legend for the published version of Fig. 2 is in error. It should be the same as that for Fig. 1.Figure2.Latencyoperatingcharacteristics(LOCs)formeanvaluesofImplicitAssociationTest(IAT)measuresforfivescoringalgorithms.MorelevelLOCcurvesindicatebetterperformance.Datapointsaremeansfor20groupsofrespondents,sortedbytheirresponsespeed.DataarefromStudy2,Election2000IATdataset.AnalyseswerelimitedtorespondentswhoindicatedstrongpreferenceforeitherBushorGoreonaself-reportitem;IATscoresforGoresupporterswerereversed.Foreachmean,nrangesbetween210and297.pts.ϭpoints.
204
GREENWALD,NOSEK,ANDBANAJI
havingalowcorrelationwithaveragelatency.Thereciprocalmeasure,whichwasbestonthecriterionoflowcorrelationwithaveragelatency,performedsopoorlyonbothelevationandlev-elnessoftheimplicit–explicitcorrelationLOC(seeFigure1)astoremoveitfromcompetitionfordesignationasthebest-performingmeasure.
Study3:PossibleRespondent-ExclusionCriteria
Instudiesthatuselatencymeasures,itisroutinetoconsiderexcludingsubjectsforeitherexcessiveslownessorexcessiveerrorrates.Forthepresentdata,itwasappropriatealsotoconsiderexclusionsforexcessivespeed,possiblyproducedbyWebsitevisitorswhowererespondingtothestimuliasrapidlyaspossiblewithouteventryingtoclassifythem.Somesuchprotocolsmightactuallyhavebeencontributedbytheresearchersortheirassoci-ates,whomighthavebeenproceedingrapidlythroughaWebIATprocedureonlyforthepurposeofcheckingitsoperation.
Method
ForeachrespondentintheElection2000dataset,anoverallmeasureofpercenterrorswascomputed,alongwiththreesummarymeasuresbasedonresponsespeed—averagelatency,percentageof“fast”(Ͻ300ms)re-sponses,andpercentageof“slow”(Ͼ3,000ms)responses.Allmeasureswerecomputedasunweightedaveragesofaveragesthatwerefirstcom-putedseparatelyforBlocks3,4,6,and7.8Eachofthefourmeasureswasinitiallyexaminedtolocatecutpointsthatwouldexclude0.25%,0.5%,0.75%,1.0%,2.5%,5.0%,and10.0%ofrespondents.Thepercentagesexcludedbythechosencutpointsdifferedslightlyfromthesetargetpercentagesbecauseofthelargenumbersoftiesinthesampleforallofthemeasuresexceptaveragelatency.Thecutpointswerethenapplied(foreachmeasureseparately)inanattempttoidentifycriteriathatwouldproduceanoticeablegaininperformanceofoneormoreofthefiveIATtransformationswhilekeepinglowthepercentageofrespondentslosttoanalysesbyexclusion.
ResultsandDiscussion
PerformancesofthefiveIATmeasures(D,mean,median,log,andreciprocal)wereexaminedintermsofeachmeasure’scorre-lationwith(a)itsparallelexplicitmeasurefortheentiresample(highvaluesaredesired)and(b)averagelatencyforthesubsampleofself-characterizedstrongsupportersofBushorGore(valuesnearzeroaredesired,indicatinglackofcontaminationofthemeasurebyslownessofresponding).
Somewhatsurprisingly,averagepercentageoffastresponseswastheonlydimensionforwhicharelativelysmallexclusionofrespondentsachievedaclearlyusefulresult.Figure3presentsthedataforcorrelationofthefiveIATmeasureswithexplicitcandi-datepreferenceasafunctionofexclusioncriteriathateliminatedsuccessivelyincreasingnumbersofrespondents.TheD,log,mean,andmedianmeasureswerearrayedinthatorder.Eachshowedmildincreasesincorrelationswithself-reportastheexclusioncriterionvariedbetweenunlimitedinclusionoffastresponses(nϭ8,218)andzerotoleranceforfastresponses(nϭ7,488,eliminating8.9%ofthesample).Bycomparisonwiththeotherfourmeasures,thereciprocalmeasureshoweddramaticimprove-mentasmorefastresponderswereexcluded,indicatingthatitsperformancewasmostimpairedbythepresenceoffastresponsesinthedataset.
Figure3.Effectsofsevencriteriaforexcludingrespondentsasafunctionoftheirproportionoffast(latencyϽ300ms)responsesoncorrelationswithself-reportforfiveImplicitAssociationTest(IAT)scoringalgo-rithms.Highercorrelationsindicatebetterperformance.Theleftmostdatapointineachcurveisfornoexclusionofrespondents.Boththeexclusioncriterionandtheremainingsamplesizeareindicatedontheabscissa.DataarefromStudy3,Election2000IATdataset.Maximumnϭ8,218.
TheDmeasure’smaximumcorrelationwithself-report(rϭ.787)wasachievedintheanalysisthatwaslimitedtorespondentswhosedatacontainednofastresponses(right-mostdatapointinFigure3).However,thisrequiredeliminating8.9%ofrespondents,whichseemedoverlycostlyinlightofthesmallgaininimplicit–explicitcorrelationbeyondthatachievedintheanalysisthatin-cludedrespondentswithupto9.5%fastresponses(rϭ.783,nϭ8,130,eliminatingonly1.1%ofrespondents).
Exclusionsbasedonaverageerrorratesalsoproducedsomeimprovementintheimplicit–explicitcorrelation.However,itwasnecessarytoeliminate9.4%ofrespondentsonthebasisoferrorratesinordertoobtainthesameimprovementachievedbyelim-inatingjust1.1%ofrespondentsonthebasisofaveragepercentageoffastresponses.Excluding9.4%ofrespondents(whichexcludedallthosewithmorethan17.5%errors)seemedanunacceptablylargelossofdata.Additionalanalysesthatconsideredexclusionsonthebasisofthecombinationofaveragepercentoffastre-sponsesandaverageerrorratesalsoprovidedinsufficientgaintojustifytheadditionallossesofdata.
Theincreaseinimplicit–explicitcorrelationforthebest-performingDmeasure—fromrϭ.773(withnoexclusion)torϭ.783(excludingrespondentswithmorethan9.5%fastrespons-es)—isnotlarge.Atthesametime,the1.5%increaseinvarianceexplained(from59.8%ϭ.7732to61.3%ϭ.7832)isnottrivial.Figure4showstheeffectsofexclusionsbasedonaveragepercentoffastresponsesonthecorrelationsofthefiveIAT
8Threeadditionalmeasureswerebasedonthemaximumpercentagesoferrors,slowresponses,andfastresponsesobservedinanysingleblock.Noneofthesemaximummeasuresprovedusefulasacriteriononwhichtobaseexclusions.Consequently,theyarenotmentionedfurther.
IMPROVEDSCORINGOFTHEIAT
205
Figure4.Effectsofsevencriteriaforexcludingrespondentsasafunctionoftheirproportionoffast(latencyϽ300ms)responsesoncorrelationswithaverageresponselatencyforfiveImplicitAssociationTest(IAT)scoringalgorithms.Lowercorrelationsindicatebetterperformance.Theleftmostdatapointineachcurveisfornoexclusionofrespondents.Boththeexclusioncriterionandtheremainingsamplesizeareindicatedontheabscissa.DataarefromStudy3,Election2000IATdataset.AnalyseswerelimitedtorespondentswhoindicatedstrongpreferenceforeitherBushorGoreonaself-reportitem;IATscoresforGoresupporterswerereversed.Maximumnϭ5,202.
measureswithaveragelatency.Thisisacorrelationforwhichthedesiredresultisclosetozero—showinglittleornocontaminationoftheIATmeasurebyresponsespeed.ThereciprocalandDmeasureswerethebestperformers,withcorrelationsuniformlybelowrϭ.10foralllevelsofexclusion.Bycomparison,thelog,median,andmeanmeasuresperformedpoorly,allhavingcorre-lationsaboverϭ.20atalllevelsofexclusion.Interestingly,theexclusionpolicybasedonaveragepercentoffastresponsesthatworkedwellforthecriterionofimplicit–explicitcorrelationsi-multaneouslyimprovedperformanceslightlyfortheDmeasure(i.e.,loweringthecorrelationwithaveragelatency)whileslightlyimpairingperformanceforthereciprocalmeasure(seeFigure4).OnthebasisofStudy3,theremainingstudiesanalyzeddatabothusingallrespondentsandeliminatingthosewithmorethan10%fastresponses.Thecriterionof10%wasselectedarbitrarilyasaroundedvalueofthe9.5%criterionthatwassuccessfullyusedfortheElection2000datasetinStudy3.
Study4:TreatmentofTrialsWithErrorResponses
Themostwidelyusedmethodofdealingwithlatenciesfromtrialswithincorrectresponsesissimplynottousethoselatencies.Researchreportsoftendescribetheproportionoftrialsonwhicherrorsoccurredandthenexcludethosetrialsfromanalysesoflatencies.Thisstrategyseemsquitesatisfactorywhen,asoftenhappens,independentvariableshavesimilareffectsonlatenciesanderrorrates.Thatis,whentreatmentsthatproducehigherresponselatenciesalsoproducehighererrorrates,analysesof
latenciesanderrorrateswillsupportthesameconclusions.Fur-thermore,becauseeffectsonerrorratesareoftenweakerthanthoseonlatencies,thestrategyofdiscardingerrorlatenciesisalsoconsideredsatisfactorywheneffectsonerrorratesareweakornonsignificant.(However,cf.Wickelgren,1977,whoquestionedthewisdomoftreatingnonsignificanterrorratedifferencesasignorable.)
Study1’sresultscallintoquestionthepracticeofroutinelydiscardingerrorlatencies.TherelevantfindingfromStudy1isthatIATmeasuresshowedhigherimplicit–explicitcorrelationswhenerrorlatencieswereincludedinanalysesthanwhentheywerediscarded.Study4wasdesignedtoconsider,asstrategiesforerrortrials,proceduresmoreelaboratethansimplyretainingordiscard-ingerrorlatencies.Thesealternativesinvolvedreplacingerrorlatencieswithvaluesthatfunctionedaserrorpenalties.
Method
AnalyseswereconductedbothonthefullElection2000datasetandonadatasetthatwasreducedbyeliminatingtherespondentsforwhommorethan10%oftrialswerefasterthan300ms(i.e.,basedontheresultsofStudy3).BecausethepreviousstudieshadclearlyestablishedthattheDmeasurewassuperiortoothertransformations(viz.,mean,median,log,andreciprocal),theanalysesinStudy4andlaterstudieswerelimitedtovariationsoftheDmeasure.
FivetypesoferrortreatmentswereevaluatedinStudy3:(a)notreat-ment—latenciesoferrorresponseswereusedinthesamefashionasthoseofcorrectresponses;(b)deletionoferrortrialsfromthedataset;(c)replacementoferrorswiththeblockmeanofcorrectresponsesplusaconstant(penalty;fivepenaltieswereused—200,400,600,800,or1,000ms);(d)replacementoferrorswiththeblockmeanofcorrectresponsesplusapenaltycomputedastheblock’sstandarddeviationofcorrectresponsesmultipliedbyaconstantof1.0,1.5,2.0,2.5,or3.0;and(e)replacementoferrorswiththeblockmeanofcorrectresponsesplusavaluecomputedastheblockmeanmultipliedby0.2,0.4,0.6,0.8,or1.0.
ThevariousstrategiesusedinStudy4rangedfromnopenaltyforerrors(i.e.,discardingerrorlatencies)topenaltiesthatwereconsiderablylargerthanthebuilt-inpenaltyprovidedbyretainingerrorlatencies.Study1hadshownthatthemeanofcorrectresponsesaveraged790ms(SDϭ301),anderrorlatenciesaveraged502msslowerthancorrectresponselatencies.Accordingly,thestrategyofretainingerrorlatencieswasapproximatelyequaltousingapenaltyinthemiddleofeachofthethreesetsoffivepenaltycomputations.
ResultsandDiscussion
Figure5showstheeffectof15error-penaltystrategiesoncorrelationoftheDmeasurewithself-reportedcandidateprefer-ence.Forcomparison,valuesfortwootherstrategies—errorla-tenciesusedwithoutalterationanderrortrialsdiscarded—areshown.Threeconclusionsareapparentfromtheplottedresults.First,andconfirmingafindingofStudy1,discardingerrortrialswasaninferiorstrategy—indeed,inferiortoall16otherstrategiesplottedinFigure5.Second,themostsuccessfulstrategywasusingunalterederrorlatencies.Third,amongthe15error-penaltyfor-mulas,mostsuccessfulwereonesthatprovidedpenaltiesthatinaveragevaluewereclosetotheaverageapproximate500-mspenaltythatresultedfromtheproceduralrequirementtoprovideacorrectresponseaftermakinganerror.
Figure6showseffectsofthe15errorpenaltiesandthetwocomparisonconditionsoncorrelationsoftheDmeasurewith
206
GREENWALD,NOSEK,ANDBANAJI
Figure5.Effectsof15strategiesforerrorpenaltiesoncorrelationswithself-reportfortheDalgorithm.Effectsofusingerrorlatenciesasisandofdeletingerrortrialsareshownaslabeledasterisks.Highercorrelationsindicatebetterperformance.DataarefromStudy4,Election2000ImplicitAssociationTestdataset,excludingrespondentswhohadmorethan10%fast(Ͻ300ms)responses.Nϭ8,132.
averagelatency.Forthismeasure,correlationsclosetozeroaredesired.Thebestresults(i.e.,smallestcorrelations)wereobtainedwitherrorpenaltiesthataddedaconstanttothemeanofcorrectresponses.Useofunalterederrorlatenciesproducedaresultthatwasneartotheresultsofdiscardingerrortrialsandusingpenaltiescomputedasaconstantproportionofthemeanofcorrectre-sponses(filledblacksquaresinFigure5).
Study4establishesthatitissatisfactorytouseunalterederrorlatenciesintheWebIAT.ThisconclusionmustbequalifiedbynotingthatintheWebIATprocedure,errorlatenciesincludedthetimerequiredtoproduceasecondresponse—ineffect,theycon-tainedabuilt-inerrorpenalty.TheconclusionfromStudy4,therefore,cannotbeextendedeitherto(a)proceduresthatdonotrequireacorrectresponseoneachtrialor(b)proceduresthatrecordthelatencytotheinitialresponse(whetherornottheerrorcorrectionisrequired).Forprocedureswithnobuilt-inerrorpen-alty,Study4indicatesthatuseofanerrorpenaltyislikelytoproducebetterresultsthanwillbeobtainedwitheitherunalterederrorlatenciesordeletionoferrortrials.However,becauseseveralerror-penaltyformulasworkedreasonablywell,theresultsofStudy4donotestablishtheclearsuperiorityofanyspecificformoferrorpenalty.ThequestionofbestformoferrorpenaltyisthereforedeferredtoStudy6,whereresultsfromallfourdatasetsarejointlyconsidered.
Study5:TreatmentsofTrialsWithExtreme(Fastor
Slow)Latencies
Inadditiontotransformationssuchaslogarithmandreciprocal,remediesforproblemsduetomisshapentailsoflatencydistribu-
tionsinclude(a)settinglowerand/orupperboundsbeyondwhichlatenciesaredeletedfromthedatasetand(b)similarly,usinglowerand/orupperboundsasvaluestowhichmoreextremevaluesarerecoded(forsimulationanalysesofmethodsfordealingwithextremelatencyvalues,seeRatcliff,1993;Miller,1994).Study5examinedbothdeletionandrecoding-to-boundarystrategies.AsinStudies3and4,performanceofIATmeasureswasevaluatedintermsofimplicit–explicitcorrelations(highervaluesdesirable)andcorrelationsoftheIATmeasurewithaveragelatency(lowervaluesdesirable).AsforStudy4,Study5waslimitedtotheDmeasurebecauseofitssuperiorperformanceinStudies1–3.
Method
Study5wasconductedasthreesubstudies.Thefirstsubstudyexamineddeletionandrecoding-to-boundaryforthelowertailofthedistribution,usingboundariesof300,350,400,450,500,or550ms.Thesecondsubstudyexamineddeletionandrecoding-to-boundaryfortheuppertail,using6,000,4,000,3,000,2,500,2,250,and2,000msasboundaries.Thefinalsubstudyexploredselectedcombinationsoflowerandupperboundaries.
ResultsandDiscussion
Figure7presentstheeffectsofthe36extreme-valuetreatmentsoncorrelationsoftheDmeasurewiththetwo-itemmeasureofexplicitcandidatepreference,Figure8presentsthecorrespondingresultsforcorrelationswithaveragelatency.Allofthesecorrela-
Figure6.Effectsof15strategiesforerrorpenaltiesoncorrelationswithaverageresponselatencyfortheDalgorithm.Effectsofusingerrorlatenciesasisandofdeletingerrortrialsareshownaslabeledasterisks.Lowercorrelationsindicatebetterperformance.DataarefromStudy4,Election2000ImplicitAssociationTest(IAT)dataset,excludingrespon-dentswhohadmorethan10%fast(Ͻ300ms)responses.AnalyseswerelimitedtorespondentswhoindicatedstrongpreferenceforeitherBushorGoreonaself-reportitem;IATscoresforGoresupporterswerereversed.Nϭ5,151.
IMPROVEDSCORINGOFTHEIAT
207
Figure7.Effectsof36strategiesfortreatinglowandhighextremelatenciesoncorrelationswithself-reportfortheDalgorithm.Thecorre-lationfordatausingnoextreme-valuetreatmentisshownasalabeledasterisk.Highercorrelationsindicatebetterperformance.Datapointstotheleftinvolvelesssevereextreme-valuetreatmentsthanthosetotheright.DataarefromStudy5,Election2000ImplicitAssociationTestdataset,excludingrespondentswhohadmorethan10%fast(Ͻ300ms)responses.Nϭ8,132.
tionswerecomputedusingthesamplethatwasreduced(fromNϭ8,218toNϭ8,132)byexcludingrespondentswhohadmorethan10%ofresponsesfasterthan300ms(onthebasisofStudy3).Inbothfigures,anasteriskshowstheresultobtainedwhennoextremevaluetreatment(beyondtheinitialdeletionoflatenciesover10,000ms)wasapplied.
Lowertailtreatments.InFigures7and8,thecurveswithopenandfilleddiamondsshow,respectively,correlationsinvolvingIATmeasuresthatusedlowertaildeletionandlowertailrecodingwithboundariesrangingfrom300to550ms.Figure7showsthatlowertaildeletion(opendiamondsinFigure7)producedsmallincreasesintheimplicit–explicitcorrelationforlowerboundaryvaluesupto450ms,abovewhichperformancewasinferiortonolowerboundtreatment.Lowerboundrecodingproducedvirtuallynochangeintheimplicit–explicitcorrelationforallsixboundaryvaluesthatwereexamined.Figure8showsthattheeffectoflowerboundtreatmentsoncorrelationswithaveragelatencywasnilforthelowesttwoboundaryvaluesforbothdeletionandrecoding.Atlowerboundariesof400msandabove,contaminationofmeasuresbyresponsespeedincreasedforthelowerbounddeletionstrategy,butnotforlowerboundrecoding.Alloftheseeffectsweresmall.Uppertailtreatments.Uppertaildeletion(curveswithopentrianglesinFigures7and8)producedaveryslightimprovementinimplicit–explicitcorrelationforthetwohighestboundaryval-ues(6,000msand4,000ms)anddeterioration(relativetonoupperboundary)atlowervaluesoftheupperbound(seeFigure7).Forallsixupperboundaryvalues,therecoding-to-boundarystrategy(filledtriangles)producedaverysmallimprovementinthe
implicit–explicitcorrelation.ForthecriterionofcorrelationoftheDmeasurewithaveragelatency,bothstrategies(deletionandrecoding-to-boundary)yieldedinferiorperformance(i.e.,highervalues)comparedtonouppertailtreatment(seeFigure8).
Combinedloweranduppertailtreatments.ThecurvesmarkedbyopenandfilledsquaresinFigures7and8showresultsforthecombinationofdeletionofvaluesbelow400mswithalloftheupperboundtreatments.InFigure7,bothdeletion(opensquares)andrecoding(filledsquares)yieldedimprovementsrel-ativetothe400-mslowerbounddeletionaloneforthetwowidestupperboundaryvalues(6,000msand4,000ms).Atnarrowervalues,thismildimprovementwasretainedfortherecodingstrat-egybutnotforthedeletionstrategy.Theresultsforthecriterionofcorrelationwithaveragelatency(Figure8)wereverysimilartothoseforuppertailtreatmentswithoutanylowertailtreatment(seeprecedingparagraph).Thatis,theseresultswereconsistentlyin-feriortousingnodeletionorrecoding(markedbytheasteriskinFigure8).
Insummary,performanceoftheDmeasurewasvirtuallyunal-teredbylowerboundrecodingatanyvalue(filleddiamondsinFigures7and8).Uppertailrecodingmodestlyimprovedimplicit–explicitcorrelationsatallupperboundvaluesbutconsistentlyincreasedcontaminationbyaverageresponselatency,asdiduppertaildeletion.Thehighestvalueoftheimplicit–explicitcorrelation(rϭ.789)occurredforthecombinationofdeletionbelow400msandrecodingvaluesabove2,500msto2,500ms.However,allof
Figure8.Effectsof36strategiesfortreatinglowandhighextremelatenciesoncorrelationswithaverageresponselatencyfortheDalgorithm.Thecorrelationfordatausingnoextreme-valuetreatmentisshownasalabeledasterisk.Lowercorrelationsindicatebetterperformance.Datapointstotheleftinvolvelesssevereextreme-valuetreatmentsthanthosetotheright.DataarefromStudy5,Election2000ImplicitAssociationTest(IAT)dataset,excludingrespondentswhohadmorethan10%fast(Ͻ300ms)responses.AnalyseswerelimitedtorespondentswhoindicatedstrongpreferenceforeitherBushorGoreonaself-reportitem;IATscoresforGoresupporterswerereversed.Nϭ5,151.
208
GREENWALD,NOSEK,ANDBANAJI
thestrategiesinvolvinguppertailtreatmentshadtheundesirableeffectofincreasingcontaminationbyaverageresponselatency(seeFigure8).Bycontrast,thestrategyoflowertaildeletionat350or400msproducedasmallimprovementinimplicit–explicitcorrelation(rϭ.785)withoutincreasing(ordecreasing)contam-inationbyaveragelatency.
InsummaryofStudy5,gainsinimplicit–explicitcorrelationresultingfromdeletionorrecodingofextremevaluesweresmall.Someofthesesmallincreaseswereaccompaniedby(undesired)increasesinthecorrelationofIATscoreswithaveragelatency.Asaconsequenceoftheseobservations,judgmentaboutthevalue(ifany)ofextreme-valuetreatmentsshouldawaitconsiderationofresultsfromStudy6,whichusedadditionalperformancecriteria.
Study6:AdditionalPerformanceCriteriaandAdditional
DataSetsSummaryofStudies1–5
Usingthecriterionofimplicit–explicitcorrelation,Study1foundthatIATmeasureswereimproved(a)slightly,byincludingthefirsttwotrialsofcombined-taskblocks(whichhadpreviouslybeendeletedfromanalyses),and(b)substantially,byincorporat-ingthedatafromtwoblocksthathadpreviouslybeentreatedaspracticetrials.Study2establishedthattheDmeasurewassuperiortoothertransformations(mean,median,log,andreciprocal)bothinmagnitudeoftheimplicit–explicitcorrelationandinminimiz-ingvariationsinthatcorrelationacrossvariationsinrespondents’averagespeedofresponding.Study2alsoshowedthattheDmeasurewassatisfactoryinhavingalowcorrelationoftheIATmeasurewithaverageresponselatency.Study3demonstratedthevalueofexcludingasmallproportionofrespondentsforwhom10%ormoreofresponseswerefasterthan300ms.Study4extendedStudy1’sfindingthatitwasusefultoretainlatenciesfromerrortrials.Thatis,Study4showedgains,relativetodeletionoferrortrials,achievedbyreplacingerrorlatencieswithvaluesthatfunctionedaserrorpenalties.Study5foundaverysmallimprovementintheDmeasurewhenresponsesfasterthan400msweredeletedfromrespondents’datasets.
ThegoalofStudy6wastoevaluateallofthescoringstrategiesthatappearedpromisinginStudies1–5.Sevenperformancecrite-riawereusedtoevaluatethesefinalists.ThefirsttwoofthesewerethetwoimportantcriteriathathadbeenusedinStudies1–5:(a)implicit–explicitcorrelationand(b)resistancetocontaminationrelatedtospeedofresponding.Theadditionalfiveperformancecriteriawere(c)internalconsistency,measuredbythecorrelationbetweenoneIATmeasurebasedonBlocks3and6andanotherbasedonBlocks4and7(seeTable1);(d)resistancetotheoften-observedordereffect(i.e.,associationsappearstrongerwhentheyaretestedinBlocks3and4ratherthaninBlocks6and7);(e)resistancetothereductioninIATscoresthatistypicallyobservedamongthosewhohavepreviouslycompletedoneormoreIATs;(f)sensitivitytomodalresponsetendencies(e.g.,theAgeIATtypicallyshowsconsiderablystrongerassociationofyoungthanoldwithpleasant);and(g)magnitudeofthestandardizedcoeffi-cientforthepathbetweenlatentimplicitandexplicitvariablesinaconfirmatoryfactoranalysis(CFA).
Method
ItwasnecessaryfirsttochoosemeasuresforinclusioninStudy6.Study2hadmadeclearthattheDmeasure,whichuseseachrespondent’slatencyvariabilitytoprovidetheunitfortheIATmeasure,decisivelyoutperformedthefourmeasuresthatwerenotsocalibrated—thatis,themeasuresbasedonthemedianlatenciesineachblock,meansofuntrans-formedlatencies,ormeansoflogarithmorreciprocaltransformations.ThissuperiorityoftheDtransformationwasasapparentintheotherthreeIATdatasets(Race,Age,andGender–Science)asitwasintheElection2000dataset.Accordingly,Studies3–6focusedonvariationsoftheDmeasure.Study4examined17strategiesfordealingwitherrortrials.Study5examined13strategiesfordealingwithextremelatenciesateachoftheupperandlowertailsoflatencydistributions.Therewere2,873(ϭ17ϫ13ϫ13)possiblecombinationsoftheseerrorandextreme-valuetreat-ments.Inaddition,Study3evaluatedeightcutpointsoneachoffourdimensionsasbasesforexcludingsubjects,alongwithanadditionaleightthatcombinedtwocriteria,foratotalof40.AddingthefouradditionalcombinationsofincludingorexcludingthefirsttwotrialsofeachblockandusingornotusingBlocks3and6,thenumberofavailablecombina-tionsofthevariationsontheDmeasurethatwereexaminedinStudies1–5approachedhalfamillion.
BecauseofthehugenumberofpossiblestrategycombinationsfortheDmeasure,itwasnecessarytoselectforStudy6aseverelyrestrictedsubset.Todothat,theauthorsconductedStudies1–5ontheremainingthreeIATdatasets(Ageattitude,Raceattitude,andGender–Sciencestereotype).Thehopewasthatthedifferentdatasetswouldreinforceeachothertoidentifyjustafewsuccessfulstrategiesfromeachstudy.Study6wouldthenexaminetheseindividuallyandincombination,withthehopethatthecombinedresultsforStudy6’ssevenperformancecriteria(describedinthethirdparagraphabove)wouldallowsettlingonone,oratmostaveryfew,variationsoftheDmeasureasanimprovedscoringalgorithmfortheIAT.OnthebasisofareviewofresultsfromthefourIATdatasets,sixvariationsoftheDmeasurewereselectedforStudy6,identifiedasD1–D6.D1wasthesimplest,involvingnoadjustmentbeyondthepreliminarydeletionoflatenciesover10,000msthatwasdoneforallmeasures.D2additionallydeletedlatenciesbelow400ms(onthebasisofStudy5).TheremainingfourDvariationsincludederrorpenalties(onthebasisofStudy4).D3replacederrortrialswiththemeanofcorrectresponsesintheblockinwhichtheerroroccurredplusapenaltyoftwicethestandarddeviationofcorrectresponsesintheblockinwhichtheerroroccurred.D4replacederrortrialswiththemeanofcorrectresponsesplus600ms.D5andD6usedthesameerrorpenaltiesasD3andD4andadditionallydeletedlatenciesbelow400ms.
Forpurposesofcomparison,Study6includedfourvariationsoftheconventionalIATmeasure,identifiedasC1–C4.C1wasthemeasureoriginallyrecommendedbyGreenwaldetal.(1998)foruseinstatisticaltests.ThismeasureuseddataonlyfromBlocks4and7(excludingtheirfirsttwotrials),recodedlatenciesoutsideboundariesof300msand3,000mstothoseboundaryvalues,andlog-transformedtheresultingvaluesbeforetakingthedifferencebetweenmeansforthetwoblocks.C2differedfromC1onlybyomittingthelogtransformation;thiswasthemeasureusedbyGreenwaldetal.(1998)forgraphicortabularpresentationofresults(becauseitsmillisecondunitsaremoreunderstandablethanthelog-transformedunits).C3usedthesamecomputationalproceduresasC1,butparalleledtheDmeasuresby(a)retainingthefirsttwotrialsofcombined-taskblocks,(b)computinganadditionalmeasureonthebasisofBlocks3and6,andthen(c)averagingthetworesultingscores.C4wasthesameasC3butomittedthelogtransformationsothatithadmillisecondunits(likeC2).
Performancesofthe10measures,D1–D6andC1–C4,wereevaluatedonthesevencriteria(seetheparagraphjustbeforethisMethodsection)forallfourIATs.Theprocedureusedtomeasureeachcriterionisdescribedtogetherwiththepresentationofitsresults.
IMPROVEDSCORINGOFTHEIAT
209
ResultsandDiscussion
Table2summarizesresultsfortheElection,Gender–Science,Race,andAgeIATsonthesevenperformancecriteria.EntriesinTable3areaveragesofthecorrespondingentriesinTable2,computedusingFisher’sr-to-Ztransformation.
Implicit–explicitcorrelation.ForallfourIATs,theexplicitmeasurewasthepreviouslydescribedone,anaveragebasedononeLikert-typeitemandtwothermometer-formatitems.AsinStudies1–5,thetwothermometeritemswerecombinedintoasinglescorebytakingtheirdifference.StandardizedtransformsofthisdifferencescoreandtheLikertitemscorewereaveragedintotheexplicit(self-report)measurethatwascorrelatedwiththe10variantsoftheIATmeasure.ThefirstdatarowsofTable2,SectionsA–D,andTable3showresultsfortheseimplicit–explicitcorrelations.WiththeloneexceptionofMeasureC3inTable2,SectionB,the6Dmeasuresoutperformedalloftheconventionalmeasuresineveryanalysis.Table3showsthatD6,whichcom-bineddeletionofvaluesbelow400mswitha600-mserrorpenalty,slightlyoutperformedtheothererror-penaltyformulas,D3,D4,andD5.Atthesametime,the2measuresthatusedunalterederrorlatencies,D1andD2,outperformedthe4Dmea-suresthatusederrorpenalties.
Resistancetocontaminationrelatedtospeedofresponding.Eachrespondent’soveralllatencyofresponsewassummarizedbycomputinganunweightedaverageofthemeanlatenciesforthefourcombined-taskblocks(Blocks3,4,6,and7).ThepossiblecontaminationofIATmeasuresbyresponse-speeddifferencesamongrespondentswasexaminedbyusingthisaveragelatencymeasuretoconstructLOCs,ofthetypesreportedpreviouslyforStudy2(seeFigures1and2).TheresultsofthoseLOCanalysesarewellrepresentedbythecorrelationsofthe10IATmeasureswithoveralllatencyaspresentedintheseconddatarowsofTable2,SectionsA–D,andTable3.Forallofthesecorrelations,thedesiredresultisrϭ0,whichwouldindicateabsenceofcontaminationoftheIATmeasurebydifferencesinoverallre-sponsespeed.TheDmeasureswereuniformlysuperiortoalloftheconventionalmeasures(i.e.,closertorϭ0)forallfourIATsindividuallyaswellasfortheiraverage,whichisshowninTable3.Inthiscase,superiorperformancewasprovidedbytwooftheDmeasuresthatincorporatederrorpenalties,D4andD6.Theirvaluesaveragedveryclosetozero.Bycomparison,forthefourconventionalmeasures,averagecorrelationsrangedbetween.157and.296,revealingasubstantiallevelofcontaminationbyindi-vidualdifferencesinresponsespeed.Thelog-transformversionsoftheconventionalprocedure(C1andC3)hadnoticeablylesscontaminationbyresponsespeedthandidthetwomeasuresthatusedmillisecondunits(C2andC4).
Internalconsistency.For8ofthe10measuresincludedinthedatacolumnsinTable2,aninternalconsistencymeasurewasprovidedbythecorrelationbetweenameasurebasedonBlocks3and6andonebasedonBlocks4and7.Thisstrategywasnotavailableforthe2conventionalmeasuresthatuseddataonlyfromBlocks4and7.Forthose2measures(C1andC2),theinternalconsistencycorrelationwascomputedasthecorrelationbetweenanIATmeasurebasedonTrials3–20ineachofBlocks4and7andasecondIATmeasurebasedonTrials21–40inthosesameblocks.Overall(seeTable6)thebest-performingmeasurewasC3,whichappliedtheconventionalIATscoringprocedurestodatafromfourblocksoftrials.AmongtheDmeasures,thetwothatdidnotuseerrorpenalties,D1andD2,producedhigherinternalconsistencycorrelationsthandidthefourthatusederrorpenalties.Somewhatsurprisingly,theseresultsindicatedthatmeasureswithrelativelypoorperformanceonthemajorcriteria(implicit–explicitcorrelationandresistancetocontaminationbyresponsespeed)hadsuperiorperformanceoninternalconsistencycorrela-tions.Thisresultsuggeststhepossibilitythatartifactualvariancecontributedtointernalconsistencyoftheconventionalmeasures.Forexample,theartifactassociatedwithaverageresponselatency(seerow2ofTable3)accountedforbetween2.5%and8.8%ofvarianceintheconventionalmeasures.Totheextentthattheconventionalmeasuresassessthisartifactreliablytheartifactwillcontributetotheirinternalconsistency,buttheresultingincreaseininternalconsistencydoesnotindicateanincreaseinvalidityofthemeasureasameasureofassociationstrength.Forthisreason,itmaybeappropriatetotreatinternalconsistencyasanuncertainguidetoconstructvalidity.
Ordereffect.TheveryfirstIATstudies(Greenwaldetal.,1998)observedeffectsoftheorderinwhichthetwopossibletaskcombinationsofeachIATwereadministered.Forexample,whenthefirsttaskinaflower–insectattitudeIATwastorespondwithoneresponsekeytoflowernamesandpleasantwordsandwiththeotherkeytoinsectnamesandunpleasantwords,performanceofthattaskwasfasterthanwhenitwasdonesecond.Thismayinvolvethefamiliarphenomenonofnegativetransfer(e.g.,Wood-worth&Schlosberg,1954),wherebypracticeatonetaskinterfereswithperformanceatasecondtaskthatrequiresgivingdifferentresponsestothefirsttask’sstimuli.Theresultofthisnegativetransferisthatthestrengthofflower–pleasantassociationsappearsgreaterwhenthetaskthatusesthisassociation—thetaskrequiringthesameresponsetoflowernamesandpleasantwords—comesfirst.
Theordereffectjustdescribedhasbeenobservedfrequentlybutnotinvariably.Ideally,anIATmeasureshouldbefreeofthisordereffect.Table2summarizesmagnitudesofobservedordereffectsinthefourdatasets.TheseareshownascorrelationsofeachIATmeasurewithadichotomousmeasureoftheorderinwhichthetwotaskswereperformed.Thedichotomousmeasurewasalwaysscoredsothattheordereffectwouldappearasapositivevalueofthiscorrelation.
ThemagnitudesofordereffectsvariedconsiderablyacrossthefourIATs.TheeffectswerenoticeablylowerfortheElectionIAT(averagerinrow4ofTable2,SectionAϭ.056)andtheRaceIAT(Table2,SectionC,averagerϭ.024)thanfortheGender–ScienceandAgeIATs(averagersϭ.278and.173,respectively,inTable2,SectionsBandD).ThesevaryingmagnitudesoftheordereffectwerealmostcertainlyduetodifferencesinproceduresamongthefourIATs.ThetwoIATswithsmallordereffectsincorporatedextratrialsineitherBlock5orBlock6oftheIATprocedure(seeTable1note).TheseextratrialsforthesecondcombinedtasklikelyovercamesomeofthenegativetransferresultingfromtasksperformedinBlocks1,3,and4.
Onaverage,theordereffectsweresimilarinmagnitudefortheDmeasuresandtheconventionalmeasures(seeTable3).How-ever,itisappropriatetolookatthedatajustforthetwoIATs(Gender–ScienceandAge)forwhichnoticeableordereffectswereobserved.Forthese(seeTable2,SectionsBandD),theDmeasuresunexpectedlyshowedsomewhatlargerordereffectsthan
210
Table2
Performanceof10MeasuresonSevenCriteria
GREENWALD,NOSEK,ANDBANAJI
VariationsofDmeasure
CharacteristicsofmeasuresIncludedtrialsLowertailtreatmentUppertailtreatmentErrortreatment
IncludeerrorlatenciesinanalysesNoneDeleteif
Ͻ400msD1D2D3D4D5D6ConventionalmeasuresC1C2ConventionalmeasureswithaddedtrialsC3C4AlltrialsofBlocks3,4,6,and7
None
DeleteifϽ400ms
Trials3–40ofAlltrialsofBlocksBlocks4and73,4,6,and7RecodelatenciesϽ300msto
300msRecodelatenciesϾ3,000msto
3,000msIncludeerrorlatenciesinanalyses
DeleteiflatencyϾ10,000ms
Replaceerrors:Replaceerrors:Replaceerrors:Replaceerrors:mean(C)mean(C)mean(C)mean(C)ϩ2SDϩ600msϩ2SDϩ600msModifiedeffectsizecomputation(seetext)
VariationsofDmeasure
LatencytransformationLogarithmNoneLogarithmConventional
measures
None
ConventionalmeasureswithaddedtrialsC3C4Sevenperformancecriteria
D1D2D3D4D5D6C1C2A:Election2000IATdataa1.2.3.4.5.6.7.
Implicit–explicitcorr..783Corr.withaveragelatency.063Internalconsistencycorr..764Ordereffectcorr..091Corr.withIATexperienceϪ.023IATeffectsize1.54Implicit–explicitpathinCFA.858
.785.066.767.086Ϫ.0271.55.860
.771.095.740.049Ϫ.0621.44.853
.773.017.747.048Ϫ.0301.46.854
.767.097.728.041Ϫ.0691.41.850
.773.019.743.039Ϫ.0361.45.854
.687.176.665.052Ϫ.0141.10.787
.663.289.636.044Ϫ.0341.01.770
.758.229.763.051Ϫ.0821.35.831
.733.364.743.043Ϫ.1131.21.810
B:Gender–ScienceIATdatab1.2.3.4.5.6.7.
Implicit–explicitcorr..251Corr.withaveragelatency.064Internalconsistencycorr..594Ordereffectcorr..251Corr.withIATexperienceϪ.094IATeffectsize1.04Implicit–explicitpathinCFA.326
.254.065.598.257Ϫ.0971.05.328
.239.056.579.296Ϫ.1001.00.311
.239.024.589.288Ϫ.0961.00.311C:RaceIATdatac1.2.3.4.5.6.7.
Implicit–explicitcorr..359Corr.withaveragelatencyϪ.018Internalconsistencycorr..564Ordereffectcorr.Ϫ.023Corr.withIATexperienceϪ.089IATeffectsize1.00Implicit–explicitpathinCFA.465
.361Ϫ.017.566Ϫ.017Ϫ.0901.00.468
.359Ϫ.010.556.039Ϫ.0960.99.467
.357Ϫ.058.558.030Ϫ.0841.00.464D:AgeIATdatad1.2.3.4.5.6.7.
Implicit–explicitcorr..170Corr.withaveragelatency.051Internalconsistencycorr..521Ordereffectcorr..127Corr.withIATexperienceϪ.204IATeffectsize1.38Implicit–explicitpathinCFA.227
.172.051.523.134Ϫ.2081.39.230
.172.042.524.197Ϫ.2001.33.231
.175Ϫ.001.527.181Ϫ.1881.34.233
.174.039.512.204Ϫ.2031.32.236
.178Ϫ.004.520.191Ϫ.1921.33.239
.106.203.574.189Ϫ.2051.08.139
.091.300.567.183Ϫ.2100.99.119
.137.204.571.150Ϫ.2501.25.177
.113.325.566.141Ϫ.2661.14.147
.360Ϫ.010.546.045Ϫ.0950.98.470
.358Ϫ.059.548.040Ϫ.0851.00.467
.292.090.579.054Ϫ.1230.82.374
.271.176.562.052Ϫ.1350.75.351
.343.105.593Ϫ.003Ϫ.1510.91.436
.322.211.580Ϫ.002Ϫ.1740.84.411
.239.055.572.302Ϫ.1020.99.316
.241.023.587.297Ϫ.1001.00.313
.196.158.598.279Ϫ.0950.81.256
.186.247.566.270Ϫ.1020.75.246
.240.168.624.261Ϫ.1170.98.304
.227.281.603.253Ϫ.1380.93.291
Note.Abbreviationsforthe10measures(D1–D6andC1–C4)areexplainedintheMethodsectionofStudy6.ThesevenperformancecriteriaaredescribedindetailintheResultssectionofStudy6.OnthebasisofStudy3,samplesexcludedrespondentsforwhommorethan10%ofIATresponseswerefasterthan300ms.mean(C)ϭblockmeanofcorrect-responselatencies;SDϭblockstandarddeviationofcorrect-responselatencies;IATϭImplicitAssociationTest;corr.ϭcorrelation;CFAϭconfirmatoryfactoranalysis.aNϭ8,132forCriteria1and7;5,151forCriteria2and6;8,784forCriteria3and4;and4,908forCriterion5.bNϭ10,475forCriteria1and7;11,549forCriteria2,3,4,and6;and10,509forCriterion5.cNϭ6,811forCriteria1and7;7,734forCriteria2,3,4,and6;and6,307forCriterion5.dNϭ10,537forCriteria1and7;11,384forCriteria2,3,4,and6;and7,194forCriterion5.
IMPROVEDSCORINGOFTHEIAT
211
Table3
Performanceof10MeasuresonSevenCriteria(AverageofFourIATs)
VariationsofDmeasure
Characteristicsofmeasure
IncludedtrialsLowertailtreatmentUppertailtreatmentErrortreatment
IncludeerrorlatenciesinanalysesNoneDeleteif
Ͻ400msD1D2D3D4D5D6ConventionalmeasuresC1C2ConventionalmeasureswithaddedtrialsC3C4AlltrialsofBlocks3,4,6,and7
AlltrialsofBlocks3,4,6,and7
None
DeleteifϽ400ms
Trials3–40ofBlocks4and7
RecodelatenciesϽ300ms
to300msRecodelatenciesϾ3,000ms
to3,000ms
DeleteiflatencyϾ10,000ms
Replaceerrors:Replaceerrors:Replaceerrors:Replaceerrors:Includeerrorlatenciesinanalysesmean(C)mean(C)mean(C)mean(C)ϩ2SDϩ600msϩ2SDϩ600msModifiedeffectsizecomputation(seetext)
VariationsofDmeasure
LogarithmNoneLogarithmNoneConventional
measures
D5.424.045.597.150Ϫ.1181.175.525
D6.428Ϫ.005.608.144Ϫ.1041.195.527
C1.347.157.605.145Ϫ.110.953.434
C2.326.254.584.139Ϫ.121.875.413
ConventionalmeasureswithaddedtrialsC3.408.177.645.116Ϫ.1511.123.491
C4.383.296.629.110Ϫ.1731.030.464
Latencytransformation
Sevenperformancecriteria1.2.3.4.5.6.7.
D1D2.436.041.624.116Ϫ.1061.248.533
D3.425.046.608.147Ϫ.1151.190.525
D4.426Ϫ.005.614.138Ϫ.1001.200.525
Implicit–explicitcorr..434Corr.withaveragelatency.040Internalconsistencycorr..621Ordereffectcorr..113Corr.withIATexperienceϪ.103IATeffectsize1.240Implicit–explicitpathinCFA.530
Note.Abbreviationsforthe10measures(D1–D6andC1–C4)areexplainedintheMethodsectionofStudy6.ThesevenperformancecriteriaaredescribedindetailintheResultssectionofStudy6.Forperformancecriterion6,entriesinthistableareaveragesofthefourcorrespondingentriesinTables2–5.Fortheremaining(correlational)criteria,entriesinthistablewerecomputedbyfirstconvertingtheentriesinTables2–5toFisher’sZandthenreconvertingtheaveragedZstor.mean(C)ϭblockmeanofcorrect-responselatencies;SDϭblockstandarddeviationofcorrect-responselatencies;IATϭImplicitAssociationTest;corr.ϭcorrelation;CFAϭconfirmatoryfactoranalysis.
theCmeasures;theDmeasuresthatusedcomputederrorpenalties(D3–D6)showedlargerordereffectsthanthosethathadbuilt-inerrorpenalties(D1andD2).TheseobservationsareconsideredfurtherintheGeneralDiscussion.
ResistancetotheeffectofpriorIATexperience.Oneoftheoptionalself-reportquestionsontheIATWebsiteaskedabouttherespondent’snumberofpriorcompletedIATs.Therewerefivereportingoptions:0,1,2,3–5,and6ormore.ItwasknownfrompreviousanalysesthatpriorexperiencewiththeIATwasassoci-atedwithareductioninIATscoresforthosewhoreportedoneormoreprioruses,comparedwiththosereportingzeroprioruses(seeGreenwald&Nosek,2001).LittleornofurtherreductioninIATscoresoccurredfortwoormoreprevioususes.Accordingly,thefive-choicemeasureofpriorIATexperiencewasreducedtoadichotomythatdistinguishedzerofromoneormoreprioruses.ItisdesirableforanIATmeasurenottobeaffectedbypreviousexperiencetakingtheIAT.TheeffectofpriorexperiencemeansthatscoresofIATnovicescannotbecompareddirectlywiththoseofnon-novicesand,forthesamereason,posttestscannotbecompareddirectlywithpretests(whenthepretestisthefirstIATtaken).ThedesiredcorrelationofanIATmeasurewiththedichot-omouspriorexperiencemeasureisthereforezero.However,theexpectationbasedonpreviousobservationsisthatthiscorrelationwillbenegative—thatis,numericallylessextremeIATscoreswillbeobservedforthosewithpriorIATexperience.ThefifthdatarowsofTable2,SectionsA–D,andTable3reportcorrelationswiththepriorexperiencemeasure.Thesecorrelationswereuniformlynegative,asexpected.ThesixDmeasuresvariedlittleandperformednoticeablybetter(i.e.,hadlowercorrelations)thanthetwoconventionalmeasuresthatuseddatafromallfourblocks(C3andC4).
Sensitivitytomodalresponsetendencies.TheAge,Race,andGender–ScienceIATstypicallyshow,respectively,strongerasso-ciationofyoungthanoldwithpleasant,strongerassociationofEuropeanAmericanthanAfricanAmericanwithpleasant,andstrongerassociationsoffemalewithartsandmalewithsciencethanoffemalewithscienceandmalewitharts.FortheElection2000IATtherewasnosimilarmodaltendencyinthepopulationofrespondents.However,therewasastrongdifferenceinIATscoresbetweenself-identified(onthe5-pointLikertitem)strongsupportersofBushandGore.Thatdifferencewasusedinthetestformodalresponsetendencies.
ThesixthdatarowsofTable2,SectionsA–D,andTable3reportthesemodaltendenciesasdeffectsizes.FortheElection2000IAT,thedmeasurederivesfromthetwo-groupcomparisonofstrongBushandstrongGoresupporters.FortheotherthreeIATs,itistheone-sampleeffectsizeoftheentiresample’sgrandmeandifferencefromzero.ThecomputationalprocedurefortheElection2000IATmadethedmeasurepartlyredundantwiththeimplicit–explicitcorrelationthatappearsinthefirstdatarowof
212
GREENWALD,NOSEK,ANDBANAJI
eachtable.However,therewasnosuchredundancyfortheotherthreeIATs.Tables2and3showthatthesixDmeasureswereconsistentlymoresensitivetomodalresponsetendenciesthanwerethefourconventionalmeasures.AmongtheDmeasures,thetwothatinvolvedonlythebuilt-inerrorpenalty(D1andD2)wereslightlysuperior,onaverage,tothefourthatusedacomputederrorpenalty.
Magnitudeofimplicit–explicitpathinCFA.Twoexplicitmeasures(Likertandthermometerdifference)wereavailableforeachIATdataset,andtwosubmeasuresofeachIAT(thetwousedintheinternalconsistencycorrelations)werealsoavailable.Thesefourmeasuresweresufficienttopermitaconfirmatoryfactoranalysis(CFA)thatusedtwomeasurestoidentifyalatentexplicitfactorandtwomeasurestoidentifyalatentimplicitfactor.Goodness-of-fitstatisticsobtainedinthevariousCFAsindicated,withoutexception,thatthistwo-factormodelfitallofthedataverywell.TheseventhdatarowofeachtableshowsthestandardizedcoefficientsforthepathbetweenlatentimplicitandexplicitfactorsobtainedfromeachoftheCFAs.Thispathcoefficientcanbeunderstoodasanestimateofthecorrelationthatmightbeobservedbetweenerror-freeimplicitandexplicitmeasures.ConsistentlyhighervaluesofthispathwereobtainedforthesixDmeasuresthanforthefourconventionalmeasures.AlthoughD1andD2wereslightlysuperiortotheotherfourDmeasures,itcanbeseeninTable3that,onaverage,therewasverylittledifferenceamongthesixDmeasures.
ComparisonofthesixDmeasures.ThemainpurposeofStudy6wastoidentifyoneormoresuperiorvariationsoftheDmeasure.ThesixDvariationsselectedforuseinStudy6variedalongtwodimensions:(a)treatmentoffastresponses(deletionoflatenciesbelow400msvs.nodeletion)and(b)treatmentoferrortrials(useoferrorlatenciesunalteredvs.replacementoferrorswiththemeanplustwicethestandarddeviationofcorrectlatenciesintheblockinwhichtheerroroccurredvs.errorreplacementbyblockmeanofcorrectlatenciesplus600ms).ThesevariationshadbeenselectedonthebasisoftheirsuperiorityoverotherstrategiesfortreatingextremelatenciesanderrorsinStudies4and5.
Tables2and3showthatthedifferencesamongthesixfinalistDmeasureswereneitherlargenorfullyconsistentacrossperfor-mancecriteriaordatasets.BecausenosingleDvariationclearlyseparateditselffromtheotherfiveinStudy6,conclusionsaboutthefeaturesthatshouldbeincludedinarevisedIATscoringalgorithmaredeferredtotheGeneralDiscussion.
GeneralDiscussion
ThepresentfindingscallstronglyforreplacingtheIAT’scon-ventionalscoringprocedure.TheconventionalIATalgorithmwasdecisivelyoutperformedbyallsixDmeasuresselectedforStudy6.ThissuperiorityoftheDmeasureswasevidentonfiveperformancecriteria:(a)magnitudeofimplicit–explicitcorrela-tion,(b)resistancetocontaminationbyresponsespeeddifferences,(c)resistancetotheIAT-score-reducingeffectofpriorexperiencewiththeIAT,(d)sensitivitytoknowneffectsonIATmeasures,and(e)latentimplicit–explicitpathinCFAs.
Thisdiscussionfocusesfirstonthepossibilityofanalternativeinterpretationoftheimportantcriterionofmagnitudeofimplicit–explicitcorrelations;nextonthetwoperformancecriteriathatdivergedfromtheotherfive—internalconsistencyandtheeffect
oforderofcombinedtasks;andthenonpracticalissuesofapply-ingthepresentresultstoresearchusesoftheIAT.
FurtherConsiderationofPerformanceCriteria
Implicit–explicitcorrelations.Implicit–explicitcorrelationswerehigherfortheDmeasuresthanforallotheralgorithms.ThisresultwasobservedconsistentlyinallfourIATdomains.Asdevelopedintheintroduction,thesehigherimplicit–explicitcor-relationscanindicategreaterconstructvalidityofanIATmeasureifassociationstrengthsareacomponentofboththeimplicitandexplicitmeasures.Thiswasillustratedintheintroductionbyanalogytothewayinwhichanimprovedmeasureofheightcanproducealargercorrelationbetweenheightandweight.Theheight–weightrelationwasproposedasanappropriateexamplebecause,conceptually,heightisacomponentofbothmeasures.Therearealsocircumstancesinwhichfindingthatamodifiedmeasureyieldsalargercorrelationwithanothermeasurecanindicatereducedconstructvalidityforthemodifiedmeasure.Sup-pose,forexample,thatmodificationofameasureofquantitativeaptitudeincreasesitscorrelationwithameasureofverbalaptitude.Thisincreasedcorrelationcouldbeduetothemodifiedquantita-tiveaptitudemeasurecontaininggreatercontaminationwithverbalaptitude.Thisstateofaffairsmightplausiblyoccurifthemodifiedquantitativemeasurehasahigherproportionofwordproblemsrelativetoproblemsrepresentedmoreabstractlywithnumbersorsymbols.Inthisverbal–quantitativeexample,thesharedcompo-nentthatincreasesthecorrelationisnotaconstruct-validaspectofquantitativeaptitudes.
Thisarticle’suseofimplicit–explicitcorrelationsaspositiveindicatorsofconstructvalidityrestsonthebeliefthatcomponentsofthesecorrelationsarebettermodeledbytheheight–weightexamplethanbytheverbal–quantitativeexample.Inorderfortheverbal–quantitativeexampletoprovidethesuperiormodel,theDmeasurewouldhavetoexceedtheotheralgorithmsincapturingsomenonassociativecomponentoftheself-reportmeasures—forexample,impressionmanagement.However,thereisnoplausiblebasisforthatconclusion.AdditionalbasisfortheconclusionthattheDtransformationissuperiorinconstructvaliditycomesfromunpublishedanalysesofotherWebIATdatasetsbythesecondauthor(Nosek,2003)showinghighercorrelationsoftheDmea-surewithseveraldemographicandsociopoliticalmeasuresthatwerehypothesizedtoberelatedtotheassociationstrengthsmea-suredbytheIAT.
Internalconsistency.Highestinternalconsistencywasunex-pectedlyobservedforMeasureC3(seethethirddatarowofTable3).OndiscoveringthisresultinStudy6,theauthorssuggestedthatthehigherinternalconsistencyofC3mightbeduetoitsbeingmorereliablysensitivethanothermeasurestoanartifactassoci-atedwithlatencydifferencesamongrespondents.Thiseffectoflatencydifferencescouldincreaseinternalconsistencywithoutcontributingtoconstructvalidity.Unfortunately,thepresentdatasetsprovidenodecisivemeansofevaluatingthisspeculation.Resistancetotheeffectoforderofcombinedtasks.ForthecriterionofresistancetotheeffectoforderofadministeringtheIAT’scombinedtasks,Study6foundthattheDmeasuresshowedlessresistancetothisundesiredeffectthandidtheconventionalmeasures.ThiswasapparentforthetwoIATs(Gender–ScienceandAge)forwhichsubstantialordereffectswereobserved(see
IMPROVEDSCORINGOFTHEIAT
213
thefourthdatarowsinTable2,SectionsBandD).Thenegativetransferinterpretationoftheordereffect(describedinStudy6)interpretstheordereffectasaninfluenceofIATproceduresonthestrengthsoftheassociationsbeingmeasured.Withthisinterpre-tation,theDmeasure’sgreaterordereffectsareconsistentwiththeDmeasure’sconstructvalidity.Nevertheless,theordereffectremainsundesirable.Fortunately,variationsinmagnitudeofordereffectsamongthefourIATsindicatethatitispossibletoavoidthisundesiredproceduralinfluenceonIATscoresbyincreasingthenumbersoftrialseitherinBlock5orBlock6oftheIATprocedure(formoreevidenceofthesuccessofthisproceduraladjustment,seeNosek,Greenwald,&Banaji,2003).
ChoiceAmongVariationsoftheDMeasure
Bysmallmargins,bestaverageperformancesonthreeofthefiveperformancecriteriathatindicatedsuperiorityoftheDmea-surewereobtainedwhenlatencieslowerthan400msweredeleted(seeMeasureD2inTable3,first,sixth,andseventhdatarows).AlthoughD2usednospecialtreatmentoferrors,ithadthesub-stantialbuilt-inerrorpenaltycreatedbytheWebIAT’srequire-menttoprovideacorrectresponseafteranyerror.ForthefourDmeasuresthatreplacederrorlatencieswithcomputedpenalties,therewasvirtuallynodifferencebetweenthetwomeasuresthatdeletedlatenciesbelow400ms(D5andD6)andthetwothatdidnot(D3andD4).Theseobservationsveryslightlyfavorthestrat-egyofdeletinglatenciesbelow400ms.However,thegainappearssoslightastomakethestrategyquestionable.
Study4hadpreviouslydemonstratedthatsuperiorresultswereachievedonthecriterionofimplicit–explicitcorrelationforthestrategyofretainingerrorlatenciescomparedwithdeletingerrortrials.Furthermore,withtheexceptionoftheGender–ScienceIAT,Study4alsofoundthatallerror-penaltyformulasyieldedhigherimplicit–explicitcorrelationsthandidthestrategyofdeletingerrortrials.Study6examinedingreaterdetailthetwoerror-penaltyformulasthathadperformedbestamongthelargernumberexam-inedinStudy4.Study6providednobasisforconcludingthateitherofthesetwoerror-penaltyformulaswassuperiortotheotherortotheprocedurallybuilt-inerrorpenalty.Rather,thebuilt-inerrorpenaltyofD1andD2wasslightlysuperiortothecalculatederrorpenalties.
TheonlyconfidentconclusionaboutpreferredformoftheDmeasuretoemergefromStudy4wasthattheDmeasureshouldbeusedwithanerrorpenalty.Theerrorpenaltymightbeabuilt-inproceduralpenalty,asforMeasuresD1orD2inStudy6.Alter-natively,forIATproceduresthatcontainnobuilt-inpenalty,eitherofthetwopenaltyformulasusedinStudy6(the600-mspenaltyorthe2ϫstandarddeviationpenalty)shouldperformapproximatelyequally.
GeneralizingtoLaboratoryUsesoftheIAT
TheanalysessummarizedinTables2and3usedsamplesthatomittedrespondentsforwhommorethan10%oftrialshadlaten-ciesfasterthan300ms.Thecutpointof10%fastresponseswasselectedasacompromiseamongcriteriathat,inStudy3,wereeffectiveintheseparateanalysesofthefourIATs.Overthefourdatasets,useofthe10%-fast-responsescutpointeliminatedanaverageof1.74%ofrespondents,whichisasmallerpercentageof
eliminationthanhasbeentypicalofmostlaboratoryIATstudies.Examinationofdataforrespondentswhohadmorethan10%fastresponsesrevealedthattheirerrorrateswereoftenhigh.Forexample,intheElection2000dataset,theaverageerrorrateforthe1.1%ofrespondentswhoexceededthe10%-fast-responsescriterionwas35.7%,comparedwithanaverageofonly8.7%errorsfortheremaining98.9%ofrespondents.
Theauthorsweresurprisedtodiscoverthatadditionalelimina-tionsbasedonhigherrorratesdidnotimproveresultsmorethanslightlybeyondwhatwasachievedwiththe10%-fast-responsescriterion.Theminoradditionalimprovementthatcouldbeachievedseemedinsufficienttojustifydiscardingarelativelylargeproportionofadditionalrespondents.Study3showedthatdiscard-ingrespondentsonthebasisofslowrespondingactuallyimpairedperformanceofthevariousIATmeasures.
The10%-fast-responsesexclusioncriterion,whichprovedmostusefulinthepresentstudies,maynotbesufficientforlaboratorystudies.Inlaboratorystudiestheremightbemorereasontodiscardrespondentsonthebasisofhigherrorratesorslowresponding.Also,inlaboratorystudiessingleaberrantcasesmayhavegreaterimpactthantheydoinverylargedatasetssuchasthoseofthepresentresearch.Itthereforeseemsunwisetousethepresentresultsasthebasisforastrongrecommendationondata-discardpoliciesforlaboratorystudies.The10%-fast-responsescriterioncanberecommendedasaminimumexclusionpolicyforlabora-torystudies.LaboratoryusersoftheIATshouldremainalertintheusualfashionforindicationsthatindividualprotocolsmaybeuntrustworthy.
TheauthorshavebeguntousetheDmeasureinlaboratoryinvestigationsinwhichtheconventionalalgorithmhasalsobeenincludedforcomparison.Theselaboratoryuseshavemostoften,butnotinvariably,indicatedlargereffectsizesfortheDmeasure.ThesevariationsinsuperiorityoftheDmeasureareconsistentwiththeexpectedvariabilityofresultsfromsmallsampleinves-tigations.OtherswillnodoubtlikewiseoccasionallyencountersamplesinwhichtheDmeasureisoutperformedwhenthesamedatasetisanalyzedwithmultiplevariationsofIATmeasures.Fortheirownresearch,theauthors’policywillbetoreportresultsfortheDmeasureregardlessofwhathasbeenfoundwithothermeasuresexaminedforcomparison.Todootherwise—forexam-ple,byselectingthemeasurethatyieldsthelargesteffectsizeonatestofinterest—willinevitablybiaseffectsizeestimates.
TheImprovedAlgorithm
TheconventionalscoringprocedureandtheimprovedalgorithmthatemergesfromthepresentanalysesarecomparedinTable4.Theimprovedalgorithmhasthreesubstantialchangesfromtheconventionalprocedure:(a)useofpractice-blockdata(Step1inTable4),(b)useoferrorpenalties(computedinSteps5and7),and(c)useofindividual-respondentstandarddeviationstopro-videthemeasure’sscaleunit(computedinStep6andappliedinStep11).
Onewaytoassessthevalueoftheimprovedalgorithmistocomputethepercentsavingsinresearchresourcesthatcanbeobtainedduetoitsexpectedeffectofincreasingresearchpower.Forthesecomputations,MeasuresD2andC1wereusedtorepre-senttheimprovedalgorithmandtheconventionalalgorithm,re-spectively.Samplesizesrequiredforpowerof.80torejectthenull
214
GREENWALD,NOSEK,ANDBANAJI
Table4
ConventionalandImprovedImplicitAssociationTest(IAT)ScoringAlgorithmsCompared
Approximatelyequivalentalternativesforimproved
algorithm
Step12
ConventionalalgorithmUsedatafromB4&B7Nonsystematiceliminationofsubjectsforexcessivelyslowrespondingand/orhigherrorratesDropfirsttwotrialsofeachblockRecodelatenciesoutside
300/3,000boundariestothenearerboundaryvalue
ImprovedalgorithmUsedatafromB3,B4,B6,&B7
Eliminatetrialswith
latenciesϾ10,000ms;eliminatesubjectsforwhommorethan10%oftrialshavelatencylessthan300msUsealltrials
Noextreme-valuetreatment(beyondStep2)ComputemeanofcorrectlatenciesforeachblockComputeonepooledSDforalltrialsinB3&B6;anotherforB4&B7Replaceeacherrorlatencywithblockmean
(computedinStep5)ϩ600ms
34567
Deletetrialswithlatenciesbelow400msAlsocomputeSDofcorrectlatenciesforeachblockComputethesepooledSDsjustforcorrectresponsesReplacementϭblockmeanϩ2ϫblockSDcomputedinStep5;
alternately,uselatencytocorrectresponseinaprocedurethatrequiresacorrectresponseafteranerror
89101112
Log-transformtheresultingvaluesAveragetheresultingvaluesforeachofthetwoblocksComputethedifference:B7ϪB4
NotransformationAveragetheresultingvaluesforeachofthefourblocks
Computetwodifferences:B6ϪB3andB7ϪB4Divideeachdifferencebyitsassociatedpooled-trialsSDfromStep6AveragethetwoquotientsfromStep11
Differencescanbe
computedintheoppositedirection
Note.Blocknumbers(e.g.,B1)refertotheproceduresequenceshowninTable1.TheconventionalalgorithmhasnoprocedurescorrespondingtoSteps5–7orSteps11–12oftheimprovedalgorithm.SDϭstandarddeviation.SPSSsyntaxforcomputingIATmeasuresusingtheimprovedalgorithmcanbeobtainedathttp://faculty.washington.edu/agg/iat_materials.htm
hypothesiswithtwo-tailed␣ϭ.05werecomputedforresearchdesignedtodeterminestatisticalsignificanceofanimplicit–explicitcorrelation.Onthebasisoftheaveragecorrelationsre-portedforMeasuresC1andD2inthefirstdatarowofTable3,theeffectsizesusedforthesepowercomputationswererϭ.347fortheconventionalalgorithmandrϭ.436fortheimprovedalgo-rithm.Cohen’s(1977,p.458)Formula10.3.5wasusedtocomputerequiredsamplesizes.9Thesecomputationsyieldedrequiredsam-plesizesof63fortheconventionalalgorithmand39fortheimprovedalgorithm.Thereductioninrequiredsamplesizeaf-fordedbytheimprovedalgorithmistherefore38.1%.Thisamountofsavingscanbeverysignificantinresearchwithhighper-respondentcosts—forexample,studiesthatuseindividual-subjectinterviewsorstudiesofdifficult-to-locatepopulations.Thesavingswouldbelargerinastudywithlowerexpectedcorrelations(e.g.,itwouldbe62.1%usingtheestimatesfromtheAgeIATasshowninTable2,SectionD).
Inadditiontothecostsavingsjustillustrated,theimprovedalgorithmoffersagaininconstructpurity.Thatis,theimprovedalgorithm,comparedwiththeconventionalscoringprocedure,islesscontaminatedbyextraneousvariables.OnesuchcontaminantistheconventionalIATmeasure’sproductionofspuriouslyex-tremeIATscoresforslowresponders(seeFigure2andsummarydataforCriterion2inTable3,MeasuresC1–C4).Thenewalgo-rithmalmostcompletelyeliminatesthisartifact(Table3,Crite-rion2,MeasuresD1–D6).Resistancetotheresponse-speedartifactshouldbeusefulinstudiesthatcompareIATscoresforgroups,suchaschildrenversusadults,thatdifferinspeedofresponding.Thenewalgorithmlikewiseshouldprovidemorevalidcorrela-tionsofIATmeasureswithindividualdifferencemeasures,suchas
Whendoingthiscomputation,Cohen’s(1977)Formula10.3.3shouldbecorrectedtoread:zЈϭarctanh(r).
9IMPROVEDSCORINGOFTHEIAT
215
ageorworkingmemorycapacity,thatcorrelatewithresponsespeed.AsecondartifactforwhichthenewalgorithmaffordssomeprotectionispriorIATexperience.CompletionofoneormoreIATstendstoreducemagnitudesofsubsequentIATscores(seeTable3,dataforCriterion5).Thenewalgorithm’sreducedsen-sitivitytopriorIATexperienceshouldbeusefulinpretest–posttestdesignsorinstudieswithmultipleIATmeasures.Unfortunately,theeffectofpriorexperienceisnotcompletelyeliminatedbythenewalgorithm(seeTable3,Criterion5,MeasuresD1–D6).Itthereforeremainsappropriate,whenusingthenewalgorithm,(a)tobecautiousininterpretingpretest–posttestdifferencesand(b)tocounterbalanceorderofadministrationformultipleIATmeasures.Thebenefitsofthenewalgorithmarenotlimitedtothefewsituationsjustillustrated.Comparedwiththepreviousconven-tionalprocedure,thenewIATalgorithmshouldgenerally(a)betterreflectunderlyingassociationstrengths,(b)morepowerfullyassessrelationsbetweenassociationstrengthsandothervariablesofinterest,(c)provideincreasedpowertoobservetheeffectofexperimentalmanipulationsonassociationstrengths,and(d)betterrevealindividualdifferencesthatareduetoassociationstrengthsratherthanothervariables.Accordingly,thenewIAT-scoringalgorithmcanberecommendedasageneralreplacementforthepreviousconventionalprocedure.
References
Asendorpf,J.B.,Banse,R.&Mu¨cke,D.(2002).Doubledissociationbetweenimplicitandexplicitpersonalityself-concept:Thecaseofshybehavior.JournalofPersonalityandSocialPsychology,83,380–393.Ashburn-Nardo,L.,Voils,C.I.,&Monteith,M.J.(2001).Implicitassociationsastheseedsofintergroupbias:Howeasilydotheytakeroot?JournalofPersonalityandSocialPsychology,81,789–799.Banse,R.,Seise,J.,&Zerbes,N.(2001).Implicitattitudestowardshomosexuality:Reliability,validity,andcontrollabilityoftheIAT.Zeitschriftfu¨rExperimentellePsychologie,48,145–160.
Brendl,C.M.,Markman,A.B.,&Messner,C.(2001).Howdoindirectmeasuresofevaluationwork?EvaluatingtheinferenceofprejudiceintheImplicitAssociationTest.JournalofPersonalityandSocialPsy-chology,81,760–773.
Brinley,J.F.(1965).Cognitivesets,speedandaccuracyofperformanceintheelderly.InA.T.Welford&J.E.Birren(Eds.),Behavior,agingandthenervoussystem(pp.114–149).Springfield,IL:CharlesCThomas.Cohen,J.(1977).Statisticalpoweranalysisforthebehavioralsciences(Rev.ed.).NewYork:AcademicPress.
Egloff,B.,&Schmukle,S.C.(2002).PredictivevalidityofanImplicitAssociationTestforassessinganxiety.JournalofPersonalityandSocialPsychology,83,1441–1455.
Faust,M.E.,Balota,D.A.,Spieler,D.H.,&Ferraro,F.R.(1999).Individualdifferencesininformation-processingrateandamount:Im-plicationsforgroupdifferencesinresponselatency.PsychologicalBul-letin,125,777–799.
Gawronski,B.(2002).WhatdoestheImplicitAssociationTestmeasure?Atestoftheconvergentanddiscriminantvalidityofprejudice-relatedIATs.ExperimentalPsychology,49,171–180.
Greenwald,A.G.(2001,October).Top10listofthingswrongwiththeIAT.PresentationattheannualconferenceoftheSocietyofExperimen-talSocialPsychology,Spokane,WA.
Greenwald,A.G.,Banaji,M.R.,Rudman,L.A.,Farnham,S.D.,Nosek,B.A.,&Mellott,D.S.(2002).Aunifiedtheoryofimplicitattitudes,stereotypes,self-esteem,andself-concept.PsychologicalReview,109,3–25.
Greenwald,A.G.,McGhee,D.E.,&Schwartz,J.L.K.(1998).Measuringindividualdifferencesinimplicitcognition:TheImplicitAssociationTest.JournalofPersonalityandSocialPsychology,74,1464–1480.Greenwald,A.G.,&Nosek,B.A.(2001).HealthoftheImplicitAssoci-ationTestatage3.Zeitschriftfu¨rExperimentellePsychologie,48,85–93.
Hummert,M.L.,Garstka,T.A.,O’Brien,L.T.,Greenwald,A.G.,&Mellott,D.S.(2002).UsingtheImplicitAssociationTesttomeasureagedifferencesinimplicitsocialcognitions.PsychologyandAging,17,482–495.
Kim,D.-Y.,&Greenwald,A.G.(1998,May).Voluntarycontrollabilityofimplicitcognition:Canimplicitattitudesbefaked?PaperpresentedattheannualmeetingoftheMidwesternPsychologicalAssociation,Chicago.
Lappin,J.D.,&Disch,K.(1972).Latencyoperatingcharacteristic:I.Effectsofstimulusprobabilityonchoicereactiontime.JournalofExperimentalPsychology,92,419–427.
Maison,D.,Greenwald,A.G.,&Bruin,R.(2001).TheImplicitAssoci-ationTestasameasureofimplicitconsumerattitudes.PolishPsycho-logicalBulletin,2,61–79.
McConnell,A.R.,&Leibold,J.M.(2001).RelationsamongtheImplicitAssociationTest,discriminatorybehavior,andexplicitmeasuresofracialattitudes.JournalofExperimentalSocialPsychology,37,435–442.
Miller,J.(1994).Effectsoftruncationonreactiontimeanalysis.JournalofExperimentalPsychology:General,123,34–80.
Nosek,B.A.(2003).[Implicitandexplicitattitudestowardsexualorien-tation].Unpublisheddata.
Nosek,B.A.,Banaji,M.R.,&Greenwald,A.G.(2002a).Harvestingimplicitgroupattitudesandbeliefsfromademonstrationwebsite.GroupDynamics,6,101–115.
Nosek,B.A.,Banaji,M.R.,&Greenwald,A.G.(2002b).Mathϭmale,meϭfemale,thereforemathme.JournalofPersonalityandSocialPsychology,83,44–59.
Nosek,B.A.,Greenwald,A.G.,&Banaji,M.R.(2003).UnderstandingandusingtheImplicitAssociationTest:II.Unpublishedmanuscript,UniversityofVirginia.
Ratcliff,R.(1993).Methodsofdealingwithreactiontimeoutliers.Psy-chologicalBulletin,114,510–532.
Ratcliff,R.,Spieler,D.,&McKoon,G.(2000).Explicitlymodelingtheeffectsofagingonresponsetime.PsychonomicBulletinandReview,7,1–25.
Rothermund,K.,&Wentura,D.(2001).Figure–groundasymmetriesintheImplicitAssociationTest(IAT).Zeitschriftfu¨rExperimentellePsy-chologie,48,94–106.
Rudman,L.A.,Feinberg,J.,&Fairchild,K.(2002).Minoritymembers’implicitattitudes:Automaticingroupbiasasafunctionofgroupstatus.SocialCognition,20,294–320.
Wickelgren,W.A.(1977).Speed–accuracytradeoffandinformationpro-cessingdynamics.ActaPsychologica,41,67–85.
Woodworth,R.S.,&Schlosberg,H.(1954).Experimentalpsychology(Rev.ed.).NewYork:Holt.
Yellott,J.I.(1971).Correctionforfastguessingandthespeed–accuracytradeoffinchoicereactiontime.JournalofMathematicalPsychology,8,159–199.
(Appendixfollows)
216
GREENWALD,NOSEK,ANDBANAJI
Appendix
QuestionsUsedtoObtainOptionalSelf-ReportMeasuresPriorto
ImplicitAssociationTest(IAT)Measures
LikertItems
One5-pointLikertitemwasusedinconjunctionwitheachIAT,illus-tratedherefortheAgeIAT:
Whichstatementbestdescribesyou?Istronglypreferyoungpeopletooldpeople.Imoderatelypreferyoungpeopletooldpeople.Ilikeyoungpeopleandoldpeopleequally.Imoderatelypreferoldpeopletoyoungpeople.Istronglypreferoldpeopletoyoungpeople.
FortheRaceIAT,theitalicizedconceptwordswerereplacedwithEuro-peanAmericansandAfricanAmericans.FortheElection2000IATtheconceptswereGeorgeW.BushandAlGore.
FortheGender–ScienceIAT,theLikertitemwasasfollows:
Whichstatementbestdescribesyou?
Istronglyassociateliberalartswithfemalesandsciencewithmales.Imoderatelyassociateliberalartswithfemalesandsciencewithmales.
Iassociatemalesandfemaleswithscienceandliberalartsequally.Imoderatelyassociatesciencewithfemalesandliberalartswithmales.
Istronglyassociatesciencewithfemalesandliberalartswithmales.
ThermometerItems
Two11-pointitemswereusedinconjunctionwitheachIAT,illustratedherefortheAgeIAT:
Pleaseratehowwarmorcoldyoufeeltowardthefollowinggroups(0ϭcoldestfeelings,5ϭneutral,10ϭwarmestfeelings).OldpeopleYoungpeople
Adrop-downlistwithnumbers0–10wasprovidedtotherightofeachofthetwoconcepts.Thethermometerscorewascomputedasthenumericaldifferencebetweenthetworesponses.FortheraceandElection2000IATs,theconceptlabelswerereplacedinthesamefashionasfortheLikertitems.
FortheGender–ScienceIAT,thethermometermeasurewasasfollows:
Pleaseratehowmuchyouassociatethefollowingdomainswithmalesorfemales.ScienceLiberalarts
Thedrop-downlisttotherightofeachofthetwoconceptsprovidedfiveoptions:stronglymale,somewhatmale,neithermaleorfemale,somewhatfemale,andstronglyfemale.Scoringthesefiveoptions,respectively,as1–5,thethermometerscorewascomputedasthenumericaldifferencebetweenthetworesponses.
ReceivedNovember7,2002RevisionreceivedMarch9,2003
AcceptedMarch25,2003Ⅲ
因篇幅问题不能全部显示，请点此查看更多更全内容