ACE = Addenbrooke’s Cognitive Examination
Theory of mind is one of the social cognition domains. It is the ability to attribute or infer beliefs, intentions, and mental states to others in order to understand and explain their behavior (Premack and Woodruff, 1978). In the past decade, theory of mind deficit has been examined in many psychiatric disorders, eg, schizophrenia, bipolar disorder, and autism spectrum disorder (Millan et al, 2012). Properly evaluating theory of mind in clinical practice and research requires valid and reliable assessment tools.
The Reading the Mind in the Eyes Test (the “Eyes Test”) (Baron-Cohen et al, 1997, 2001) is one of the most widely used theory of mind measures. Participants are asked to look at 36 pictures of the eye regions of men’s and women’s faces, each picture accompanied by four adjectives. The task is to choose the one word that best describes what the person in the picture is thinking or feeling. Previous studies have shown that the test can distinguish many populations with psychiatric disorders, most often people with schizophrenia or autism spectrum disorder, from healthy control participants (eg, Baron-Cohen et al, 1997; Savla et al, 2012).
The Eyes Test has been translated into languages such as German (Pfaltz et al, 2013), French (Prevost et al, 2014), and Italian (Vellante et al, 2013). A Thai version, however, has not been available before now. Our objective in this study was to evaluate the validity and reliability of our Thai translation of the Eyes Test.
Before recruiting participants, we determined how big a sample would be required. Based on a correlation sample size, we found that we would need a minimum of 52 participants to achieve 80% power to detect a correlation of 0.38 between the Eyes Test score and another measure of emotional recognition, according to Alaerts et al (2011).
During the months of December 2015 through April 2016, we recruited 70 participants: 30 people with schizophrenia and 40 normal controls. To be able to assess the test in populations with known groups validity, we enrolled the people with schizophrenia from a mental health clinic at Thammasat University Hospital in Pathumthani, Thailand. Their schizophrenia had been diagnosed by certified psychiatrists using the criteria of the Diagnostic and Statistical Manual of Mental Disorders, 5th ed (American Psychiatric Association, 2013).
We recruited the controls from among patients or relatives of patients at other clinics within the hospital. The controls had no history of schizophrenia, bipolar disorder, major depressive disorder, intellectual disability, or any other major neurologic or psychiatric condition.
All participants were literate in the Thai language and had at least 6 years of formal education.
The study was approved by the Human Ethics Committee of Thammasat University (Protocol Number MTU-EC-PS-6-151/58). All participants gave informed consent.
Thai-Language Translation of the Eyes Test
As mentioned, the Eyes Test is a 36-item measure originally developed by Baron-Cohen et al (1997, 2001). Each of us (psychiatrist authors T.C. and T.L.) independently translated the test, with its crucial descriptive adjectives, from the original English language into Thai. We compared and discussed our “forward” translations and agreed on a first combined version. Then a professional translator translated our version backward into English. We compared our first version and the backward translation to identify problematic items and reach consensus on revisions. Last, we ensured that the correct Thai-language answer for each picture had the same meaning as the English word.
We used the Faces Test (Baron-Cohen et al, 1997) to examine convergent validity. We gave our Thai translation of the test, which we have validated but not yet published. The Faces Test also measures recognition of emotions. Participants view 20 photographs of full faces showing a variety of emotional expressions. In this test, each picture is accompanied by two adjectives. Participants are asked to choose the one that describes the face. Scores range from 0 to 20. A higher score indicates better emotion recognition. We gave the Faces Test to evaluate convergent validity with the Eyes Test, and we expected to find a correlation.
Addenbrooke’s Cognitive Examination-III
The Addenbrooke’s Cognitive Examination-III (ACE) (Hsieh et al, 2013) is a 100-point assessment of overall function in five neurocognitive domains: attention, verbal fluency, language, visuospatial ability, and memory. Scores range from 0 to 100, with higher scores indicating better neurocognitive function (Charernboon et al, 2016; Hsieh et al, 2013). We gave the validated Thai version of the ACE (Charernboon et al, 2016) to evaluate convergent validity with the Eyes Test, again expecting to see a correlation.
All testing was done at Thammasat University Hospital. Participants first completed a demographic data form, which included their sex, age, and level of education. Research assistants, who were blinded to whether the participants were in the schizophrenia or control group, then gave them the ACE, the Faces Test, and the Eyes Test. So that we could assess test-retest reliability, 25 participants, selected by convenience sampling, took the tests twice within 2 weeks.
We used the Pearson correlation to measure the convergent validity of the Eyes Test compared to the Faces Test and the ACE. We used an independent t test to assess the known groups validity between the participants with schizophrenia and the controls. We measured internal consistency with the Cronbach alpha. We calculated test-retest reliability with the Spearman correlation and intraclass correlation coefficient. We used the Spearman correlation for test-retest reliability because the data had a non-normal distribution.
We used STATA Statistical Software Version 14 (StataCorp, College Station, Texas) to analyze our data. We considered a P value of 0.05 to be statistically significant.
As shown in Table 1, our 70 participants (80% women) had a mean age of 36.5 (standard deviation=11.3) years and an education level of 13.5 (standard deviation=4.0) years. We found no significant age or education differences between the schizophrenia and control groups.
For known groups validity, the mean scores of the participants with schizophrenia and the controls were statistically different (Table 1).
Regarding convergent validity, the correlation between the Faces Test and the Eyes Test was significant (r=0.51, P<0.001). The Eyes Test also correlated with the ACE (r=0.68, P<0.001).
Internal consistency by the Cronbach alpha for the Eyes Test was 0.7, and test-retest reliability, as measured by the Spearman correlation, was 0.95 (P<0.001). The intraclass correlation coefficient was 0.92 (95% confidence interval: 0.7-0.99, P<0.001).
As an additional test of agreement between test and retest, we constructed a Bland-Altman plot, shown in Figure 1. The Bland-Altman plot represents every difference between test and retest scores against the average of the measurements and shows an estimated agreement interval. We found that the 95% limit for the range of possible error was (−3.5, 2.7), 0% outside the limits of agreement.
This study assessed the validity and reliability of our Thai-language version of the Reading the Mind in the Eyes Test (the “Eyes Test”). To our knowledge, this is the first validated measure that assesses theory of mind in Thailand. Our results provide a framework for understanding the psychometric properties of the instrument.
Our control group’s scores on the Thai version were slightly lower than those reported by Baron-Cohen et al (2001) for the original English version: 24.6 (standard deviation=3.9) versus 26.2 (standard deviation=3.6) points. However, our control group’s scores were similar to those reported for other translated versions. For example, the German version (Pfaltz et al, 2013) had an average score of 24.5 (standard deviation=3.5), and the Italian version (Vellante et al, 2013), 24.8 (standard deviation=4.2).
Our controls scored higher on the Eyes Test than our participants with schizophrenia, thus supporting the known groups validity of the measure. These results agree with many studies showing theory of mind deficits in people with schizophrenia (eg, de Achával et al, 2010; Savla et al, 2012).
In this study, we demonstrated a moderate correlation between the Eyes Test and the Faces Test. Both of these measures evaluate the social cognition domain, but using different subdomains. Both require basic emotion recognition abilities; in theory, therefore, their results should correlate to some degree. Although the Eyes Test was not directly compared to the Faces Test in previous validation studies, much research has shown that the Eyes Test usually correlates with other emotion recognition tests. For example, Alaerts et al (2011) found a significant correlation between performance on the Eyes Test and the Emotion Recognition Test (r=0.38; P=0.03). This moderate correlation illustrated that the tests share the same general, but not identical, concepts.
Adequate performance on the Eyes Test also requires somewhat intact neurocognition. Thus, scores usually correlate with neurocognitive function. Our study demonstrated a moderate correlation between the Eyes Test and the ACE (r=0.68). de Achával et al (2010) also showed a moderate correlation between the Eyes Test and the ACE (r=0.44, P=0.05). Many studies have found an association between the Eyes Test and tests of intelligence quotient (Ahmed and Miller, 2011; Stanford et al, 2011).
Our results show acceptable internal consistency for the Eyes Test. In most previously published studies, the reported internal consistency reliability was poor to acceptable, with the reliability being between 0.5 and 0.7 (Girli, 2014; Prevost et al, 2014; Vellante et al, 2013).
Several attributes of the Eyes Test can affect its internal consistency. For example, some of the very difficult-to-interpret pictures have a correct response rate of about 50%, but some relatively easy pictures have a correct response rate above 80%. The accuracy of responses can also be affected by the sex of the person pictured and the angle at which the face is shown, as well as by the commonness of the words offered to describe the emotion. All of these variables may reduce the test’s internal consistency. Our study confirms what other available research has suggested, that the Eyes Test does not have excellent internal consistency and homogeneity.
We found an excellent test-retest reliability of 0.95. This accords with other validation studies, most of which report test-retest reliability of at least 0.6 (Hallerbäck et al, 2009; Prevost et al, 2014; Vellante et al, 2013).
Our sample of normal controls may not have been large enough to establish a baseline or norms value for the general population. Further studies should include a large participant base with a wide range of ages and education levels to provide reference values. We could not determine convergent validity with other theory of mind measures because they do not have available Thai-language versions.
We also note the significant difference in the numbers of women and men in our schizophrenia and control groups. However, several theory of mind studies in patients with schizophrenia and normal controls have shown that one’s sex did not predict performance on the Eyes Test (eg, de Achával et al, 2010; Savla et al, 2012).
In summary, our findings show that the Thai version of the Reading the Mind in the Eyes Test is a valid and reliable measure of theory of mind and is easy to give and score. The test can be used in both clinical practice and future investigations of theory of mind in neurologic and psychiatric disorders.
The authors thank the anonymous reviewers for their insightful comments.
Ahmed FS, Miller LS. 2011. Executive function mechanisms of theory of mind
. J Autism Dev Disord. 41:667–678.
Alaerts K, Nackaerts E, Meyns P, et al. 2011. Action and emotion recognition from point light displays: an investigation of gender differences. PloS One. 6:e20989.
American Psychiatric Association. 2013. Diagnostic and Statistical Manual of Mental Disorders, 5th ed. Washington DC: American Psychiatric Association.
Baron-Cohen S, Wheelwright S, Hill J, et al. 2001. The “Reading the Mind in the Eyes
” test revised version: a study with normal adults, and adults with Asperger syndrome or high-functioning autism. J Child Psychol Psychiatry. 42:241–251.
Baron-Cohen S, Wheelwright S, Jolliffe T. 1997. Is there a “language of the eyes”? Evidence from normal adults, and adults with autism or Asperger syndrome. Visual Cognition. 4:311–331.
Charernboon T, Jaisin K, Lerthattasilp T. 2016. The Thai Version of the Addenbrooke’s Cognitive Examination III. Psychiatry Investig. 13:571–573.
de Achával D, Costanzo EY, Villarreal M, et al. 2010. Emotion processing and theory of mind
patients and their unaffected first-degree relatives. Neuropsychologia. 48:1209–1215.
Girli A. 2014. Psychometric properties of the Turkish child and adult form of “Reading the Mind in the Eyes
Test.” Psychology. 5:1321–1337.
Hallerbäck MU, Lugnegård T, Hjärthag F, et al. 2009. The Reading the Mind in the Eyes
Test: test–retest reliability of a Swedish version. Cogn Neuropsychiatry. 14:127–143.
Hsieh S, Schubert S, Hoon C, et al. 2013. Validation of the Addenbrooke’s Cognitive Examination III in frontotemporal dementia and Alzheimer’s disease. Dement Geriatr Cogn Disord. 36:242–250.
Millan MJ, Agid Y, Brüne M, et al. 2012. Cognitive dysfunction in psychiatric disorders: characteristics, causes and the quest for improved therapy. Nat Rev Drug Discov. 11:141–168.
Pfaltz MC, McAleese S, Saladin A, et al. 2013. The Reading the Mind in the Eyes
Test: test–retest reliability and preliminary psychometric properties of the German version. Int J Adv Psychol. 2:1–9.
Premack D, Woodruff G. 1978. Does the chimpanzee have a theory of mind
? Behav Brain Sci. 1:515–526.
Prevost M, Carrier M-E, Chowne G, et al. 2014. The Reading the Mind in the Eyes
test: validation of a French version and exploration of cultural variations in a multi-ethnic city. Cogn Neuropsychiatry. 19:189–204.
Savla GN, Vella L, Armstrong CC, et al. 2012. Deficits in domains of social cognition
: a meta-analysis of the empirical evidence. Schizophr Bull. 39:979–992.
Stanford AD, Messinger J, Malaspina D, et al. 2011. Theory of Mind
in patients at clinical high risk for psychosis. Schizophr Res. 131:11–17.
Vellante M, Baron-Cohen S, Melis M, et al. 2013. The “Reading the Mind in the Eyes
” test: systematic review of psychometric properties and a validation study in Italy. Cogn Neuropsychiatry. 18:326–354.