Secondary Logo

Share this article on:

Spanish Linguistic Validation of the Velopharyngeal Insufficiency Effects on Life Outcomes

VELO-Spanish

Skirko, Jonathan R., MD, MHPA, MPH*; Santillana, Rosario M., BA; Roth, Christina T., MS, CCC-SLP; Dunbar, Chance, BA; Tollefson, Travis T., MD, MPH

Plastic and Reconstructive Surgery – Global Open: November 2018 - Volume 6 - Issue 11 - p e1986
doi: 10.1097/GOX.0000000000001986
Original Article
United States

Background: Quality of life (QOL) assessments are useful tools that measure a patient’s health status and monitor patient-reported outcome measures. This study highlights the process of linguistic validation of a QOL assessment to serve Spanish-speaking families and ultimately help decrease language barriers in the treatment of velopharyngeal insufficiency (VPI).

Methods: The standardized linguistic validation process included forward and backward translation, reconciliation, and cognitive interviews with patients and families. Preliminary instrument test-retest measurement was assessed. Eligibility for cognitive interviews included families with familiarity of velopharyngeal insufficiency. Exclusion criteria included illiteracy and parent and child respondents who do not speak Spanish. Reliability was tested by intraclass correlation (ICC) on VPI Effects on Life Outcomes (VELO)–Spanish instruments completion on 2 measurements from the medical record.

Results: The instrument was optimized through a standardized forward and backward translation process. Further problematic language was identified during cognitive interviews with families and their children. In the second interview, only minimal changes were needed. Twenty-one patients (8 males and 13 females) were included. Mean (SD) age was 8.0 (5.3) years (range, 3–21 years). The mean (SD) Velo-Spanish score was 65 (22.1); range 32.7–100. The VELO-Spanish instrument demonstrated excellent test–retest reliability [ICC = 0.91; n = 21 and internal consistency (α = 0.96)].

Conclusions: The Spanish VELO has been developed and refined for use in Spanish-speaking populations as a VPI-specific QOL instrument. The linguistic validation process including cognitive interviews and initial reliability testing. The instrument may improve the understanding of patient-reported outcomes and potential disparities from linguistic and cultural barriers in VPI treatment

From the *Division of Otolaryngology, University of Utah, Salt Lake City, Utah

University of California, Davis Department of Otolaryngology-Head and Neck Surgery, Sacramento, Calif.

Published online 15 November 2018.

Received for publication August 17, 2018; accepted September 5, 2018.

Disclosure: The authors have no financial interest to declare in relation to the content of this article. The Article Processing Charge was paid for by the authors.

Travis T. Tollefson, MD, MPH, Facial Plastic and Reconstructive Surgery, Department of Otolaryngology-Head and Neck Surgery, UC Davis Medical Center, 2521 Stockton Blvd., Suite 7200, Sacramento, CA 95817, E-mail: tttollefson@ucdavis.edu

This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

Back to Top | Article Outline

INTRODUCTION

The Spanish-speaking population is rapidly growing in the United States. It is projected that nearly 20% of the U.S. population will prefer Spanish language by 2050.1 Language has been widely determined as a crucial component influencing the quality of health care and may also play an important role in exacerbating health care disparities among racial or ethnic groups.2 Limited English proficiency negatively impacts the quality and continuity of primary care that Latino patients receive.2 The effects of limited English proficiency have been shown to be similar in specialty care, although the impact of language barriers in this area has not been as well explored. Addressing language barriers for Spanish-speaking patients and families can reduce the health disparities they face and help optimize care for a substantial part of the patient population in this country.1 , 3

Velopharyngeal insufficiency (VPI) is a disorder that results in inadequate closure of the velopharyngeal sphincter, which requires proper functioning of the velum (soft palate) and lateral and posterior pharyngeal walls.4 VPI is characterized by hypernasal speech/resonance, nasal air emission, and occasionally nasal reflux of swallowed food and liquids.5 In addition, VPI can significantly limit communication and thus severely affect patients’ lives.4 VPI is most commonly associated with cleft palate and it is estimated that 20–40% of patients will exhibit VPI requiring surgical intervention after a cleft palate repair.6

There is a lack of literature assessing health disparities related to VPI. However, there is a small body of research dedicated to this topic in the context of cleft care. Racial and ethnic disparities exist in cleft care as fewer Hispanic children receive timely primary treatment than their White/non-Hispanic counterparts.7 , 8 Delay of care is a marker of disparity and could lead to longer term consequences for a child’s wellbeing. Delays in cleft palate repair are associated with an increased likelihood of developing VPI.9 Further, ethnicity is associated with delay in other secondary cleft surgeries.10 Although delays in care are important proxies for disparities, their true importance is in the impact they have on a child’s daily life.

Quality of life (QOL) assessments can be used to understand the patient’s health status and to monitor patient health outcomes.11 To measure the specific way VPI affects children’s lives, the VPI Effects on Life Outcomes (VELO) instrument was developed and validated.12 , 13 Children with VPI were found to have significant impairment in their QOL and had significant improvement with speech surgery.14–16 There is currently no available translation of the VELO instrument for Spanish-speaking patients and families with limited English proficiency. To address this significant language barrier in VPI care, this article describes the linguistic validation of this disease-specific QOL instrument into Spanish.

Although it may seem simple to translate an instrument into a new target language, a linguistic validation involves an iterative process so that the translation is conceptually equivalent to the primary instrument.17 In other words, the linguistic validation ensures that the translated items tap into the same concept as originally intended.18 This process is important because poorly translated instruments can threaten the validity of any research using them. It can be challenging to assess both the validity and conceptual equivalence of translations that do not follow this rigorous a process.17 The objective of this study was to create a linguistically and culturally valid Spanish VELO instrument.

Back to Top | Article Outline

MATERIALS AND METHODS

Instrument

The VELO instrument is available as either a parent or child report (indicated for children older than 8 years of age). The VELO was developed using focus groups with patients and families and expert review to provide content validity of the instrument.15 The VELO categorizes items into 6 distinct domains: speech limitation, swallowing problems, situational difficulty, emotional impact, perception by others, and caregiver impact. Both the parent and child report include the same domains, with the exception of caregiver impact which is not included in the child report. The VELO parent report consists of a total of 26 items, and the VELO child report consists of 23 items. Each item is scored on a Likert-type scale, ranging from never (zero) to almost always (4), and total scores range from 0 to 100 with 100 representing a high QOL score.

Back to Top | Article Outline

LINGUISTIC VALIDATION OVERVIEW

The linguistic validation method followed the guidelines of the International Society for Pharmacoeconomics and Outcomes Research that included forward translations, a backward translation, and cognitive interviewing.17 , 19 The instrument first underwent forward translations by 2 independent native Spanish speakers, which were then reconciled by comparing similarities and discrepancies during an in-person meeting. Afterward, a third, blinded translator completed a backward translation from Spanish back to English. The final backward translation in English was compared with the original English VELO, and the differences were reconciled to ensure conceptual equivalence between the translation and original instrument. Based on the reconciliation of the backward English VELO instrument, a second version of the Spanish translation of the VELO was created.

Group cognitive interviews were completed with a panel of parents and patients with VPI using the second version of the Spanish VELO instrument. A native Spanish speaker also fluent in English led the interviews with scripted questions as the participants interpreted the translated Spanish VELO item by item. Alterations were made on a consensus basis. The open discussion of the cognitive interviewing led to the final version of the Spanish VELO. A target reading level of third grade for the Youth Report and sixth grade for the Parent Report was established based on the Flesch-Kincaid readability formula.

Lastly, the VELO-Spanish instrument was introduced to all children with VPI with Spanish as their native language. The measurement of two consecutive instruments were retrospectively reviewed to evaluate the test–retest reliability.

Back to Top | Article Outline

Translation Process

Forward Translations

The first step in the linguistic validation was to produce 2 independent forward translations into the target (Spanish) language. Two translators, both native Spanish speakers and fluent in English, independently produced a forward translation of the VELO instrument into the target language. Both the parent report (VELO-P) and the youth report (VELO-Y) were translated, as were the instruction sheets. The 2 translations were then reconciled into a single translation, the first version of the Spanish VELO. To reconcile the independent forward translations most effectively, the translators were required to discuss their respective translations, and any discrepancies between the translations, in person.

Back to Top | Article Outline

Backward Translation

The reconciled forward translation, or first version of the Spanish VELO, was then translated back into English by a third translator. This translation is referred to as the backward translation. The third translator had no previous access to the original English VELO, which allowed the translator to be blinded. The third translator was also both a native Spanish speaker and fluent in English. The backward translation and the original English VELO were then compared by the research team. This allowed the research team to discuss the intended meaning and purpose of each VELO item and make appropriate changes to the backward translation. During the review process, the team detected and dealt with discrepancies or linguistic alternatives. Attention was also paid to the reading level of the instruments with a target grade level of third grade for the youth version (VELO-Y) and sixth grade for the parent version (VELO-P). The overarching goal of all agreed upon changes was to make the translation in the target language conceptually equivalent to the English VELO, and thus allow the carryover of the English validation to the new target language. The discussion and modifications of the backward translation was used to finalize the preinterview version of the Spanish VELO.

Back to Top | Article Outline

Cognitive Interviewing

The preinterview version of the Spanish VELO was administered to a panel of respondents using cognitive interview techniques as approved by the UC Davis Medical Center Institutional Review Board. Parents eligible to participate in the cognitive interview included those with a child diagnosed with or treated for VPI at the UC Davis Medical Center Otolaryngology Department. Exclusion criteria included illiteracy, as parents and children were required to read all question items, non-native Spanish speaker status, and age less than 8 years old for child respondents. Cognitive interviews were conducted until minimal changes were made to the Spanish VELO translation with a goal of 3–4 participants in each cognitive interview.

The respondents were interviewed as a group by the senior author (T.T.) and R.S., a native Spanish speaker also fluent in English. During the interview, respondents were asked to read the items of the Spanish VELO assessment, interpret the meaning of each item, and provide possible alternatives for confusing translations. Each cognitive interview included a discussion of both the parent and youth versions of the Spanish VELO. Parent and child respondents were asked to actively participate and were prompted with questions to determine whether each item was cognitively equivalent, or conceptually similar in meaning and purpose, to the original VELO. Audio for each cognitive interview was recorded and a report was compiled of participant feedback. Feedback from each cognitive interview was used to make changes to the translation. For an item to be considered for modification, more than one parent/child had to agree the language needed changing. During this process, linguistic issues that were addressed included question length, vocabulary (ie, ensuring that certain words could be understood by Spanish speakers from different countries with distinct dialects), and difficulty (ie, including vocabulary that was readable by parents with an elementary school grade level). This was repeated in an iterative process creating the final version of the Spanish VELO. The final version was then implemented into clinical practice just as the English version.

Back to Top | Article Outline

VELO SPANISH RELIABILITY TESTING

After institutional review board approval, the authors performed a retrospective chart review of patients with VPI from the University of California Davis Cleft Team between June 2017 and May 2018. Inclusion criteria included children 3–21 years of age with VPI and cleft palate (with or without cleft lip) with Spanish as the primary language in the home. Exclusion criteria included severe cognitive impairment that would affect communication or development, lack of follow-up, or patients who underwent VPI surgery between their initial and follow-up questionnaires. Data points included age and sex. Subjects completed the Spanish version of the VELO instrument on 2 occasions as part of typical patient-reported outcome measurement. Parents completed the Parent Report for subjects of all ages, and children aged 8 years or older, who were able to read Spanish, completed the Youth Report. The follow-up (retest) questionnaires were completed at routine clinic visits. The interval between the test–retest was chosen during an absence of treatment (no surgery or significant speech therapy) to best achieve a test–retest of the instrument. The time intervals minimize immediate recall by being greater than 3 weeks.

Back to Top | Article Outline

STATISTICAL ANALYSIS OF RELIABILITY

Internal consistency of the baseline scores of VELO Parent total and each of the 6 subscales (Speech, Swallow, Situational Difficulty, Emotional, Perception, and Caregiver Impact) was assessed using Cronbach’s alpha. A Cronbach’s alpha score greater than 0.70 is considered sufficient based on previously accepted standards.20 Test–retest reliability between baseline scores and follow-up scores was analyzed with the intraclass correlation coefficient (ICC). Correlations greater than 0.60 were determined to be adequate.21 Statistical analysis was performed using Stata (version 12; Stata-Corp, College Station, Tex.).

Back to Top | Article Outline

RESULTS

Translation

The purpose of this process was to minimize linguistic and cultural gaps in the finalized Spanish VELO translation. Through these efforts, the Spanish VELO is both comprehensible and cognitively equivalent to the English language VELO instrument. During the process of forward and backward translation, several problematically worded items were identified and optimized. Examples are found in Table 1.

Table 1

Table 1

Back to Top | Article Outline

Cognitive Interviews

Seven participants (4 parents and 3 children) were included in 2 cognitive interviews. Children with VPI had an age range of 8–12. Of these children, 1 had untreated VPI, and 2 after secondary speech surgery for VPI (Furlow palatoplasty and sphincter pharyngoplasty, respectively).

The first cognitive interview identified items that consistently caused difficulty for our parent and child respondents. Table 2 provides examples of scripted questions that were used throughout the cognitive interviews. With some questions/items, prompts were used by the interviewer to engage the participants in the discussion. Feedback and alternative translations suggested by participants were recorded, and consensus was obtained from the expert panel and the modified VELO instrument was administered in subsequent interviews. During the second cognitive interview, only minimal problems were identified so the process of cognitive interviewing was concluded. Examples of modifications to the instrument based on the cognitive interviews are compiled in Table 3.

Table 2

Table 2

Table 3

Table 3

Back to Top | Article Outline

RELIABILITY

VELO-Spanish test–retest was collected on a total of 21 patients (8 males, 13 females), and 20 parents. The mean time (SD) interval between test–retest was 74.0 (33.1) days (range, 24–168 days). Mean patient (SD) age was 8.0 (5.3) years with range (3–21 years). One young adult (20-year-old) patient completed the VELO without a corresponding parent report. The mean (SD) VELO-Spanish score was 65 (22.1); range 32.7–100. The VELO Parent total demonstrated excellent test–retest reliability (ICC = 0.91) and internal consistency (α = 0.95). Each of the VELO Parent subscales, except Emotional Impact, also demonstrated an adequate test–retest reliability and internal consistency (Table 4).

Table 4

Table 4

Back to Top | Article Outline

DISCUSSION

This study followed a linguistic validation process that included forward and backward translations, cognitive interviews, and reliability testing. This rigorous methodology helped ensure conceptual equivalence between the original instrument and its translation, so that the intended meaning of each question was maintained. In addition to thorough review by research staff and translators, this study involved Spanish-speaking patients and families through cognitive interviews, which served to further improve the conceptual equivalence of the translation.

Despite a meticulous translation process including review by expert panel of the forward and back translations, the inconsistencies identified during the cognitive interviews highlight the need for this step during rigorous linguistic validation. The cognitive interviews included patients and families who would be completing the VELO instrument as a typical clinical practice, allow us to ensure that the language chosen resonates with their personal experiences, dialect, and culture differences. The cognitive interview process helps to identify these discrepancies, while ensuring conceptual equivalence. It also helps to ensure the grade level and associated comprehension is appropriate for the target population.

Previous reports have suggested that the modifications of an existing patient-reported outcome instrument, such as presented here in the Spanish translation of the VELO), may only require qualitative evaluation of the instrument rather than more comprehensive repeat validation.22 The original VELO validation process provided content validity, while creating a variety of target language versions is being pursued in Turkish, Mandarin, French, and other dialects of Spanish.

The process outlined in this study highlights that optimizing a translation to the appropriate reading grade level is an ongoing challenge. For several items in the translation, the reading grade level was difficult to ascertain, hindering efforts to confirm that the translation was at the appropriate overall reading grade level. Reading grade level in the English version was determined by the Flesch-Kincaid readability formula, which uses number of words in a sentence and number of syllables per word as metrics for readability.23 Because the number of syllables in a word is a poor indicator of readability in the Spanish language, the Flesch-Kincaid readability formula may not be as helpful as in English.24 , 25 This provides further support of using cognitive interviews to assess readability. Special attention was paid during cognitive interviews to items with potentially higher reading grade level to help ensure comprehension. Although the cognitive interviewing showed that the translations were comprehensible, this highlights an underlying problem in wording the translated items. Special attention should be paid to these items in future translations of the VELO into new target languages.

The use of group interviews has an advantage over individual interviews because it allows participants to discuss potential problematic wording among themselves.26 This discussion helps to ensure that one participant’s impression/confusion of a question/item doesn’t inappropriately alter the instrument. This facilitates faster resolution of problematic questions/items. The benefit of individual interviews includes an open discussion of sensitive topics. Because the content of the VELO did not include sensitive topics, group interviews were chosen. The size and composition of interviews can also vary from study to study. We chose a smaller group size to allow all participants (parents and children) ample time to discuss the items. There is a possibility that larger groups require more time to involve all members, which may result in participant fatigue if not handled appropriately.

Cognitive interview participants were selected to ensure broad experiences and age ranges of the VPI patients, a recommendation of guidelines for translation and linguistic validation. This helps ensure the initial validation carries over to the target population and culture.17 Our sample additionally included adults (18–21 years old) to begin to provide adults with VPI a patient-reported outcome. Because most centers are able to treat VPI before reaching adulthood, this population is limited. Future work extending the VELO instrument to adult self-report will be of importance, especially in underserved populations. The limited number of cognitive interviews could also be considered a limitation of the study. There is no consensus in the literature on the number of respondents needed for these cognitive interviews. However, as only minimal problems were identified in the second cognitive interview, finalizing the instrument after 2 interviews was considered appropriate. Previous translation projects of the VELO instrument have not specifically documented involving patients and families native to the target language in this way.25

Our sample of Spanish-speaking and Spanish-reading children over age 8 was limited. This limited the sample size available for the VELO – Youth. Discussion with these children from Spanish speaking families revealed that many children speak Spanish as their primary language at home but were not literate in written Spanish. These children preferred to complete the written English self-report VELO. For populations with children that prefer written Spanish language to written English language, further validation of the VELO – Youth is needed.

The translation was found to be reliable in our small sample by both internal consistency and test–retest reliability. Reliability is an important piece of instrument assessment. Reliability is especially important for instruments that are used to measure change with an intervention, such as VPI surgery. The reliability of the Spanish translation is similar to that identified in the validation of the English language VELO.12–14 The reliability of the emotional impact subscale was below our threshold and below that identified in the English validation, which may be due to a relatively small sample size and a complex topic to measure. The Emotional Impact subscale seeks to capture information on being teased, feelings of sadness, frustration, or shyness due to speech difficulties. It is possible that psychosocial resources that may have been provided to subjects between their test and retest. The preliminary instrument testing is reassuring, but more analysis is needed to ensure the validation of the VELO has been retained in this Spanish translation.

The finalized version of the Spanish VELO was intended to develop a conceptually equivalent, linguistically correct, and easily comprehensible instrument.27 This was in large part successful for the VELO-Parent instrument. The VELO-Youth version was not testing in this setting as it was not found to be useful with children who preferred to read in English. Additional data collection and modifications for Spanish dialect and geographic differences are necessary and ongoing.

Back to Top | Article Outline

CONCLUSIONS

This study creates a Spanish translation of a VPI-specific QOL instrument through a well-established linguistic validation process including forward and backward translations, cognitive interviews, and reliability testing. The Spanish translation of this effective QOL instrument can be utilized in an underserved patient population, helping to minimize the linguistic and cultural barriers in the treatment of VPI and reduce possible health disparities. The success of this translation can also serve as a template for researchers hoping to achieve this goal for other non–English-speaking patient populations.

Back to Top | Article Outline

ACKNOWLEDGMENT

The VELO-Spanish instrument is available without charge via e-mail to the authors.

Back to Top | Article Outline

REFERENCES

1. Eamranond PP, Davis RB, Phillips RS, et al. Patient-physician language concordance and lifestyle counseling among Spanish-speaking patients. J Immigr Minor Health. 2009;11:494–498.
2. DuBard CA, Gizlice Z. Language spoken and differences in health status, access to care, and receipt of preventive services among US Hispanics. Am J Public Health. 2008;98:2021–2028.
3. Jaramillo J, Snyder E, Dunlap JL, et al. The Hispanic clinic for pediatric surgery: a model to improve parent–provider communication for Hispanic pediatric surgery patients. J Pediatr Sur. 2016:670–674.
4. Conley SF, Gosain AK, Marks SM, et al. Identification and assessment of velopharyngeal inadequacy. Am J Otolaryngol. 1997;18:38–46.
5. Kummer A. Cleft palate and craniofacial anomalies: effects on speech and resonance. Delmar: Cengage Learning. 2014198–199.
6. Caldarelli DD. Incidence and type of otopathology associated with congenital palatopharyngeal incompetence. Laryngoscope. 1978;88:1970–1982.
7. Cassell CH, Daniels J, Meyer RE. Timeliness of primary cleft lip/palate surgery. Cleft Palate Craniofac J. 2009;46:588–597.
8. Abbott MM, Kokorowski PJ, Meara JG. Timeliness of surgical care in children with special health care needs: delayed palate repair for publicly insured and minority children with cleft palate. J Pediatr Surg. 2011;46:1319–1324.
9. Marrinan EM, LaBrie RA, Mulliken JB. Velopharyngeal function in nonsyndromic cleft palate: relevance of surgical technique, age at repair, and cleft type. Cleft Palate Craniofac J. 1998;35:95–100.
10. Silvestre J, Basta MN, Fischer JP, et al. Minority and public insurance status: is there a delay to alveolar bone grafting surgery? Cleft Palate Craniofac J. 2017;54:e1–e6.
11. Fitzpatrick R, Fletcher A, Gore S, et al. Quality of life measures in health care. I: applications and issues in assessment. BMJ. 1992;305:1074–1077.
12. Skirko JR, Weaver EM, Perkins J, et al. Modification and evaluation of a velopharyngeal insufficiency quality-of-life instrument. Arch Otolaryngol Head Neck Surg. 2012;138:929–935.
13. Skirko JR, Weaver EM, Perkins JA, et al. Validity and responsiveness of VELO: a velopharyngeal insufficiency quality of life measure. Otolaryngol Head Neck Surg. 2013;149:304–311.
14. Skirko JR, Weaver EM, Perkins JA, et al. Change in quality of life with velopharyngeal insufficiency surgery. Otolaryngol Head Neck Surg. 2015;153:857–864.
15. Barr L, Thibeault SL, Muntz H, et al. Quality of life in children with velopharyngeal insufficiency. Arch Otolaryngol Head Neck Surg. 2007;133:224–229.
16. Bhuskute A, Skirko JR, Roth C, et al. Association of velopharyngeal insufficiency with quality of life and patient-reported outcomes after speech surgery. JAMA Facial Plast Surg. 2017; 19(5):406–412. doi: 10.1001/jamafacial.2017.0639.
17. Wild D, Grove A, Martin M, et al; ISPOR Task Force for Translation and Cultural Adaptation. Principles of good practice for the translation and cultural adaptation Process for Patient-Reported Outcomes (PRO) measures: report of the ISPOR task force for translation and cultural adaptation. Value Health. 2005;8:94–104.
18. Herdman M, Fox-Rushby J, Badia X. ‘Equivalence’ and the translation and adaptation of health-related quality of life questionnaires. Qual Life Res. 1997;6:237–247.
19. Canales S, Ganz PA, Coscarelli CA. “Translation and validation of a quality of life instrument for Hispanic American cancer patients: methodological considerations.” Qual Life Res. 1995;4(1):3–11.
20. Terwee CB, Bot SD, de Boer MR, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60:34–42.
21. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174.
22. Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims: draft guidance. Health Qual Life Outcomes. 2006;4:79.
23. Kincaid JP, Fishburne RP, Rogers RL, et al. Derivation of new readability formulas (automated readability index, fog count, and Flesch reading ease formula) for Navy enlisted personnel. Research Branch Report 8–75. Chief of Naval Technical Training: Naval Air Station Memphis (1975).
24. Gilliam B, Pena SC, Mountain L. The fry graph applied to Spanish readability. Read Teach. 1980;33:426–430. JSTOR. Available at www.jstor.org/stable/20195034.
25. Bruneel L, Van Lierde K, Bettens K, et al. Health-related quality of life in patients with cleft palate: validity and reliability of the VPI Effects on Life Outcomes (VELO) questionnaire translated to Dutch. Int J Pediatr Otorhinolaryngol. 2017;98:91–96.
26. Epstein J, Santo RM, Guillemin F. A review of guidelines for cross-cultural adaptation of questionnaires could not bring out a consensus. J Clin Epidemiol. 2015;68:435–441.
    27. Santillana R, Skirko J, Roth C, et al. Spanish linguistic validation for the velopharyngeal insufficiency effects on life outcomes. JAMA Facial Plast Surg. 2018;20:331–332.
    Copyright © 2018 The Authors. Published by Wolters Kluwer Health, Inc. on behalf of the American Society of Plastic Surgeons. All rights reserved.