The impact of extrapolation and examiner’s judgment on Rorschach form quality coding: Interrater reliability and clinical implications

Davide Ghirardello, Francesca Ales, Valeria Raimondi, Donald J. Viglione, Alessandro Zennaro, Luciano Giromini

Accepted December 31, 2020

First published December 31, 2020

https://doi.org/10.26387/bpa.289.2

Abstract

Rorschach Form Quality (FQ) describes how well a response fits a given inkblot location and is derived from how frequently it is identified and whether raters judge it to be a good fit. A large body of research has established that FQ has excellent validity as a measure of reality testing and severity of psychological disturbance. However, some studies have reported sub-optimal interrater reliability (IRR). In this article we inspected 1588 responses from 60 archival
Rorschach protocols. We examined the frequency with which FQ was Tabled (T), Extrapolated (E) or Judged (J), and tested the strength of the association of FQ determination path to FQ scores and IRR. Results showed that when compared to T responses, E and J responses were characterized by increasingly poorer FQ and less optimal IRR. Taken together, these results confirm that the determination path used to code FQ has a notable impact on the scoring
accuracy and IRR of FQ. In order to reduce the FQ coding inconsistencies, the authors suggest that future R-PAS developments might try to incorporate computer algorithms to help with the attribution of FQ codes.

References

  • ACKLIN, M.W., McDOWELL, C.J., VERSCHELL, M.S. & CHAN, D.(2000). Interobserver agreement, intraobserver reliability, andthe Rorschach Comprehensive System. Journal of PersonalityAssessment, 74, 15-47.

    doi.org/10.1207/S15327752JPA740103
  • BECK, S.J., BECK, A.G., LEVITT, E.E. & MOLISH, H.B. (1961).Rorschach’s test: I. Basic processes (3rd ed.). Grune & Stratton.

  • BERKOWITZ, M. & LEVINE, J. (1953). Rorschach scoring categoriesas diagnostic “signs”. Journal of Consulting Psychology, 17, 110-112.

    doi.org/10.1037/h0062113
  • BERRY, B.A. & MEYER, G.J. (2019). Contemporary data on thelocation of response objects in Rorschach’s inkblots. Journal ofPersonality Assessment, 101 (4), 402-413.

    doi.org/10.1080/00223891.2017.1408016
  • BURIN, D., PIGNOLO, C., ALES, F., GIROMINI, L., PYASIK,M., GHIRARDELLO, D., ZENNARO, A., ANGILETTA, M.,CASTELLINO, L. & PIA, L. (2019). Relationships betweenpersonality features and rubber hand illusion: An explorativestudy. Frontiers in Psychology.

    doi.org/10.3389/fpsyg.2019.02762
  • CICCHETTI, D.V. (1994). Guidelines, criteria, and rules of thumbfor evaluating normed and standardized assessment instrumentsin psychology. Psychological Assessment, 6, 284-290.

    doi.org/10.1037/1040-3590.6.4.284
  • COHEN, J. (1960). A coefficient of agreement for nominal scales.Education and Psychological Measurement, XX (1), 37-46.

    doi.org/10.1177%2F001316446002000104
  • DAO, T.K., PREVATT, F. & HOME, H.L. (2008). Differentiatingpsychotic patients from nonpsychotic patients with the MMPI-2and Rorschach. Journal of Personality Assessment, 90, 93-101.

    doi.org/10.1080/00223890701693819
  • EXNER, J.E. (1969). The Rorschach Systems. Grune & Stratton.

  • EXNER, J.E. (1974). The Rorschach: A comprehensive system. JohnWiley & Sons.

  • EXNER, J.E. (1996). Critical bits and the Rorschach responseprocess. Journal of Personality Assessment, 67 (3), 464-477.

    doi.org/10.1207/s15327752jpa6703_3
  • EXNER, J.E. (2003). The Rorschach: A comprehensive system, Vol. 1:Basic foundations (4th ed.). Wiley.

  • GOLDFRIED, M.R. (1962). Rorschach developmental level and theMMPI as measures of severity of psychological disturbance.Journal of Projective Techniques, 26, 187-192.

    doi.org/10.1080/08853126.1962.10381095
  • HARDER, D.W. & RITZLER, B.A. (1979). A comparison ofRorschach developmental level and 51 form-level systems asindicators of psychosis. Journal of Projective Techniques, 43, 347-354.

    doi.org/10.1207/s15327752jpa4304_2
  • KIMHY, D., CORCORAN, C., HARKAVY-FRIEDMAN, J.M.,RITZLER, B., JAVITT, D.C. & MALASPINA, D. (2007). Visualform perception: A comparison of individuals at high risk forpsychosis, recent onset schizophrenia and chronic schizophrenia.Schizophrenia Research, 97, 25-34.

    doi.org/10.1016/j.schres.2007.08.022
  • KIVISALU, T.M., LEWEY, J.H., SHAFFER, T.W. & CANFIELD, M.L.(2016). An investigation of interrater reliability for the RorschachPerformance Assessment System (R-PAS) in a nonpatient U.S.sample. Journal of Personality Assessment, 98 (4), 382-390.

    doi.org/10.1080/00223891.2015.1118380
  • KLOPFER, B., AINSWORTH, M.D., KLOPFER, W.G. & HOLT,R.R. (1954). Developments in the Rorschach technique. Vol. 1.Technique and theory. World Book Co.

  • LEWEY, J.H., KIVISALU, T.M. & GIROMINI, L. (2018). Coding withR-PAS: Does prior training with the exner comprehensive systemimpact interrater reliability compared to those examiners withonly R-PAS-Based training? Journal of Personality Assessment,101 (4), 393-401.

    doi.org/10.1080/00223891.2018.1476361
  • MEYER, G.J., HILSENROTH, M.J., BAXTER, D., EXNER JR,J.E., FOWLER, J.C., PIERS, C.C. & RESNICK, J. (2002).An examination of interrater reliability for scoring theRorschach comprehensive system in eight data sets. Journalof Personality Assessment, 78 (2), 219-274.

    doi.org/10.1207/S15327752JPA7802_03
  • MEYER, G.J., SHAFFER, T.W., ERDBERG P. & HORN S.L. (2015).Addressing issues in the development and use of the compositeinternational reference values as Rorschach norms for adults.Journal of Personality Assessment, 97 (4), 330-347.

    doi.org/10.1080/00223891.2014.961603
  • MEYER, G.J., VIGLIONE, D.J., MIHURA, J.L., ERARD, R.E. &ERDBERG, P. (2011). Rorschach Performance Assessment System:Administration, coding, interpretation and technical manual.Rorschach Performance Assessment System.

  • MIHURA, J.L. & MEYER, G.J. (Eds.). (2018). Using the RorschachPerformance Assessment System (R-PAS). The Guilford Press.

  • MIHURA, J.L., MEYER, G.J., DUMITRASCU, N. & BOMBEL,G. (2013). The validity of individual Rorschach variables:Systematic reviews and meta-analyses of the comprehensivesystem. Psychological Bulletin, 139 (3), 548-605.

    doi.org/10.1037/a0029406
  • PIANOWSKI, G., MEYER, G.J. & DE VILLEMOR-AMARAL, A.E.(2016). The impact of R-Optimized administration modelingprocedures on Brazilian normative reference values forRorschach scores. Journal of Personality Assessment, 98 (4), 408-418.

    doi.org/10.1080/00223891.2016.1148701
  • PIGNOLO, C., GIROMINI, L., ANDO’, A., GHIRARDELLO, D.,DI GIROLAMO, M., ALES, F. & ZENNARO, A. (2017). Aninterrater reliability study of Rorschach Performance AssessmentSystem (R-PAS) raw and complexity-adjusted scores. Journal ofPersonality Assessment, 99 (6), 619-625.

    doi.org/10.1080/00223891.2017.1296844
  • PIGNOLO, C., VIGLIONE, D.J. & GIROMINI, L. (2021). Howreliably can examiners make Form Quality (FQ) judgments inthe absence of the Form Quality (FQ) tables? Rorschachiana, 42,21-34.

    doi.org/10.1027/1192-5604/a000135
  • RORSCHACH, H. (1921). Psychodiagnostik. Hans Huber.

  • SHROUT, P.E. & FLEISS, J.L. (1979). Intraclass correlations: Uses inassessing rater reliability. Psychological Bulletin, 86 (2), 420-428.

    doi.org/10.1037//0033-2909.86.2.420
  • SU, W.-S., VIGLIONE, D.J., GREEN, E.E., TAM, W.-C. C., SU, J.-A.& CHANG, Y.-T. (2015). Cultural and linguistic adaptability ofthe Rorschach Performance Assessment System as a measureof psychotic characteristics and severity of mental disturbancein Taiwan. Psychological Assessment, 27 (4), 1273-1285.

    doi.org/10.1037/pas0000144
  • VIGLIONE, D.J. (2002). Rorschach coding solutions: A reference guidefor the comprehensive system. Donald J. Viglione.

  • VIGLIONE, D.J. (2010). Rorschach coding solutions: A referenceguide for the comprehensive system (2nd ed.). www.rorschachcodingsolutions.com

  • VIGLIONE, D.J., BLUME-MARCOVICI, A.C., MILLER, H.L.,GIROMINI, L. & MEYER, G. (2012). An interrater reliabilitystudy for the Rorschach Performance Assessment System.Journal of Personality Assessment, 94 (6), 607-612.

    doi.org/10.1080/00223891.2012.684118
  • VIGLIONE, D.J. & MEYER G.J. (2008). An overview of Rorschachpsychometrics for forensic practice. In C.B. Gacono & F.B. Evanswith N. Kaser-Boyd (Eds.), Handbook of forensic Rorschachpsychology. Lawrence Erlbaum Associates.

  • VIGLIONE, D., MEYER, G., MIHURA, J., ERARD, B., ERDBERG,P. & GIROMINI, L. (2016). Guidance for coding form qualityrequiring “Judgment of fit” and an important supplement on FQcoding for former CS users. www.r-pas.org.

  • VIGLIONE, D.J., MEYER, G.J., RESENDE A.C. & PIGNOLO, C.(2017). A survey of challenges experienced by new learnerscoding the Rorschach. Journal of Personality Assessment, 99 (3),315-323.

    doi.org/10.1080/00223891.2016.1233559
SHOW ALL REFERENCES (35)HIDE REFERENCES

Article info

Issue:

Keywords:

Views:

508

Downloads:

60

Cite the article:

Author Surname Author Initial. Title. Publication Title. Year Published;Volume number(Issue number):Pages Used. doi:DOI Number.


Ghirardello Davide. Ales Francesca. Raimondi Valeria. Viglione Donald J.. Zennaro Alessandro. Giromini Luciano. The impact of extrapolation and examiner’s judgment on Rorschach form quality coding: Interrater reliability and clinical implications. BPA Applied Psychology Bulletin. 2020;289(1):10-24. doi:10.26387/bpa.289.1.

Citation tool

How to cite this article

Author Surname Author Initial. Title. Publication Title. Year Published;Volume number(Issue number):Pages Used. doi:DOI Number.


Ghirardello Davide. Ales Francesca. Raimondi Valeria. Viglione Donald J.. Zennaro Alessandro. Giromini Luciano. The impact of extrapolation and examiner’s judgment on Rorschach form quality coding: Interrater reliability and clinical implications. BPA Applied Psychology Bulletin. 2020;289(1):10-24. doi:10.26387/bpa.289.1.