The impact of extrapolation and examiner’s judgment on Rorschach form quality coding: Interrater reliability and clinical implications

Davide Ghirardello, Francesca Ales, Valeria Raimondi, Donald J. Viglione, Alessandro Zennaro, Luciano Giromini

Accepted December 31, 2020

First published December 31, 2020

https://doi.org/10.26387/bpa.289.2

Abstract

Rorschach Form Quality (FQ) describes how well a response fits a given inkblot location and is derived from how frequently it is identified and whether raters judge it to be a good fit. A large body of research has established that FQ has excellent validity as a measure of reality testing and severity of psychological disturbance. However, some studies have reported sub-optimal interrater reliability (IRR). In this article we inspected 1588 responses from 60 archival
Rorschach protocols. We examined the frequency with which FQ was Tabled (T), Extrapolated (E) or Judged (J), and tested the strength of the association of FQ determination path to FQ scores and IRR. Results showed that when compared to T responses, E and J responses were characterized by increasingly poorer FQ and less optimal IRR. Taken together, these results confirm that the determination path used to code FQ has a notable impact on the scoring
accuracy and IRR of FQ. In order to reduce the FQ coding inconsistencies, the authors suggest that future R-PAS developments might try to incorporate computer algorithms to help with the attribution of FQ codes.

References

ACKLIN, M.W., McDOWELL, C.J., VERSCHELL, M.S. & CHAN, D.(2000). Interobserver agreement, intraobserver reliability, andthe Rorschach Comprehensive System. Journal of PersonalityAssessment, 74, 15-47.
doi.org/10.1207/S15327752JPA740103
BECK, S.J., BECK, A.G., LEVITT, E.E. & MOLISH, H.B. (1961).Rorschach’s test: I. Basic processes (3rd ed.). Grune & Stratton.
BERKOWITZ, M. & LEVINE, J. (1953). Rorschach scoring categoriesas diagnostic “signs”. Journal of Consulting Psychology, 17, 110-112.
doi.org/10.1037/h0062113
BERRY, B.A. & MEYER, G.J. (2019). Contemporary data on thelocation of response objects in Rorschach’s inkblots. Journal ofPersonality Assessment, 101 (4), 402-413.
doi.org/10.1080/00223891.2017.1408016
BURIN, D., PIGNOLO, C., ALES, F., GIROMINI, L., PYASIK,M., GHIRARDELLO, D., ZENNARO, A., ANGILETTA, M.,CASTELLINO, L. & PIA, L. (2019). Relationships betweenpersonality features and rubber hand illusion: An explorativestudy. Frontiers in Psychology.
doi.org/10.3389/fpsyg.2019.02762
CICCHETTI, D.V. (1994). Guidelines, criteria, and rules of thumbfor evaluating normed and standardized assessment instrumentsin psychology. Psychological Assessment, 6, 284-290.
doi.org/10.1037/1040-3590.6.4.284
COHEN, J. (1960). A coefficient of agreement for nominal scales.Education and Psychological Measurement, XX (1), 37-46.
doi.org/10.1177%2F001316446002000104
DAO, T.K., PREVATT, F. & HOME, H.L. (2008). Differentiatingpsychotic patients from nonpsychotic patients with the MMPI-2and Rorschach. Journal of Personality Assessment, 90, 93-101.
doi.org/10.1080/00223890701693819
EXNER, J.E. (1969). The Rorschach Systems. Grune & Stratton.
EXNER, J.E. (1974). The Rorschach: A comprehensive system. JohnWiley & Sons.
EXNER, J.E. (1996). Critical bits and the Rorschach responseprocess. Journal of Personality Assessment, 67 (3), 464-477.
doi.org/10.1207/s15327752jpa6703_3
EXNER, J.E. (2003). The Rorschach: A comprehensive system, Vol. 1:Basic foundations (4th ed.). Wiley.
GOLDFRIED, M.R. (1962). Rorschach developmental level and theMMPI as measures of severity of psychological disturbance.Journal of Projective Techniques, 26, 187-192.
doi.org/10.1080/08853126.1962.10381095
HARDER, D.W. & RITZLER, B.A. (1979). A comparison ofRorschach developmental level and 51 form-level systems asindicators of psychosis. Journal of Projective Techniques, 43, 347-354.
doi.org/10.1207/s15327752jpa4304_2
KIMHY, D., CORCORAN, C., HARKAVY-FRIEDMAN, J.M.,RITZLER, B., JAVITT, D.C. & MALASPINA, D. (2007). Visualform perception: A comparison of individuals at high risk forpsychosis, recent onset schizophrenia and chronic schizophrenia.Schizophrenia Research, 97, 25-34.
doi.org/10.1016/j.schres.2007.08.022
KIVISALU, T.M., LEWEY, J.H., SHAFFER, T.W. & CANFIELD, M.L.(2016). An investigation of interrater reliability for the RorschachPerformance Assessment System (R-PAS) in a nonpatient U.S.sample. Journal of Personality Assessment, 98 (4), 382-390.
doi.org/10.1080/00223891.2015.1118380
KLOPFER, B., AINSWORTH, M.D., KLOPFER, W.G. & HOLT,R.R. (1954). Developments in the Rorschach technique. Vol. 1.Technique and theory. World Book Co.
LEWEY, J.H., KIVISALU, T.M. & GIROMINI, L. (2018). Coding withR-PAS: Does prior training with the exner comprehensive systemimpact interrater reliability compared to those examiners withonly R-PAS-Based training? Journal of Personality Assessment,101 (4), 393-401.
doi.org/10.1080/00223891.2018.1476361
MEYER, G.J., HILSENROTH, M.J., BAXTER, D., EXNER JR,J.E., FOWLER, J.C., PIERS, C.C. & RESNICK, J. (2002).An examination of interrater reliability for scoring theRorschach comprehensive system in eight data sets. Journalof Personality Assessment, 78 (2), 219-274.
doi.org/10.1207/S15327752JPA7802_03
MEYER, G.J., SHAFFER, T.W., ERDBERG P. & HORN S.L. (2015).Addressing issues in the development and use of the compositeinternational reference values as Rorschach norms for adults.Journal of Personality Assessment, 97 (4), 330-347.
doi.org/10.1080/00223891.2014.961603
MEYER, G.J., VIGLIONE, D.J., MIHURA, J.L., ERARD, R.E. &ERDBERG, P. (2011). Rorschach Performance Assessment System:Administration, coding, interpretation and technical manual.Rorschach Performance Assessment System.
MIHURA, J.L. & MEYER, G.J. (Eds.). (2018). Using the RorschachPerformance Assessment System (R-PAS). The Guilford Press.
MIHURA, J.L., MEYER, G.J., DUMITRASCU, N. & BOMBEL,G. (2013). The validity of individual Rorschach variables:Systematic reviews and meta-analyses of the comprehensivesystem. Psychological Bulletin, 139 (3), 548-605.
doi.org/10.1037/a0029406
PIANOWSKI, G., MEYER, G.J. & DE VILLEMOR-AMARAL, A.E.(2016). The impact of R-Optimized administration modelingprocedures on Brazilian normative reference values forRorschach scores. Journal of Personality Assessment, 98 (4), 408-418.
doi.org/10.1080/00223891.2016.1148701
PIGNOLO, C., GIROMINI, L., ANDO’, A., GHIRARDELLO, D.,DI GIROLAMO, M., ALES, F. & ZENNARO, A. (2017). Aninterrater reliability study of Rorschach Performance AssessmentSystem (R-PAS) raw and complexity-adjusted scores. Journal ofPersonality Assessment, 99 (6), 619-625.
doi.org/10.1080/00223891.2017.1296844
PIGNOLO, C., VIGLIONE, D.J. & GIROMINI, L. (2021). Howreliably can examiners make Form Quality (FQ) judgments inthe absence of the Form Quality (FQ) tables? Rorschachiana, 42,21-34.
doi.org/10.1027/1192-5604/a000135
RORSCHACH, H. (1921). Psychodiagnostik. Hans Huber.
SHROUT, P.E. & FLEISS, J.L. (1979). Intraclass correlations: Uses inassessing rater reliability. Psychological Bulletin, 86 (2), 420-428.
doi.org/10.1037//0033-2909.86.2.420
SU, W.-S., VIGLIONE, D.J., GREEN, E.E., TAM, W.-C. C., SU, J.-A.& CHANG, Y.-T. (2015). Cultural and linguistic adaptability ofthe Rorschach Performance Assessment System as a measureof psychotic characteristics and severity of mental disturbancein Taiwan. Psychological Assessment, 27 (4), 1273-1285.
doi.org/10.1037/pas0000144
VIGLIONE, D.J. (2002). Rorschach coding solutions: A reference guidefor the comprehensive system. Donald J. Viglione.
VIGLIONE, D.J. (2010). Rorschach coding solutions: A referenceguide for the comprehensive system (2nd ed.). www.rorschachcodingsolutions.com
VIGLIONE, D.J., BLUME-MARCOVICI, A.C., MILLER, H.L.,GIROMINI, L. & MEYER, G. (2012). An interrater reliabilitystudy for the Rorschach Performance Assessment System.Journal of Personality Assessment, 94 (6), 607-612.
doi.org/10.1080/00223891.2012.684118
VIGLIONE, D.J. & MEYER G.J. (2008). An overview of Rorschachpsychometrics for forensic practice. In C.B. Gacono & F.B. Evanswith N. Kaser-Boyd (Eds.), Handbook of forensic Rorschachpsychology. Lawrence Erlbaum Associates.
VIGLIONE, D., MEYER, G., MIHURA, J., ERARD, B., ERDBERG,P. & GIROMINI, L. (2016). Guidance for coding form qualityrequiring “Judgment of fit” and an important supplement on FQcoding for former CS users. www.r-pas.org.
VIGLIONE, D.J., MEYER, G.J., RESENDE A.C. & PIGNOLO, C.(2017). A survey of challenges experienced by new learnerscoding the Rorschach. Journal of Personality Assessment, 99 (3),315-323.
doi.org/10.1080/00223891.2016.1233559

SHOW ALL REFERENCES (35)HIDE REFERENCES

DOWNLOAD PDF

Issue:

Issue 289

Keywords:

Views:

511

Downloads:

Cite the article:

Ghirardello Davide. Ales Francesca. Raimondi Valeria. Viglione Donald J.. Zennaro Alessandro. Giromini Luciano. The impact of extrapolation and examiner’s judgment on Rorschach form quality coding: Interrater reliability and clinical implications. BPA Applied Psychology Bulletin. 2020;289(1):10-24. doi:10.26387/bpa.289.1.

CITE THE ARTICLE

3332

The impact of extrapolation and examiner’s judgment on Rorschach form quality coding: Interrater reliability and clinical implications

Abstract

References

Article info

Issue:

Keywords:

Views:

Downloads:

Cite the article:

BPA - BULLETIN OF APPLIED PSYHOLOGY

MANAGEMENT AND EDITORIAL OFFICE

ABOUT US

Citation tool

How to cite this article