Share this article on:

Afterword: Lessons Learned About Multicenter Research Collaboration

Moeller, Mary Pat1; Tomblin, J. Bruce2; the OCHL Collaboration

doi: 10.1097/AUD.0000000000000215
Research Articles

In this brief afterword, we discuss the challenges and lessons learned in the process of implementing a multisite, longitudinal study. Some of the lessons learned by the research team are shared regarding research design and analysis, strategies implemented to reduce threats to validity, and techniques used to promote teamwork and collaboration across sites.

In this brief afterword, we discuss the challenges and lessons learned in the process of implementing a multisite, longitudinal study. Some of the lessons learned by the research team are shared regarding research design and analysis, strategies implemented to reduce threats to validity, and techniques used to promote teamwork and collaboration across sites.

1Center for Childhood Deafness, Boys Town National Research Hospital, Omaha, Nebraska, USA; 2Department of Communication Sciences and Disorders, University of Iowa, Iowa City, Iowa, USA; 3University of Iowa, Iowa City, Iowa, USA; 4University of North Carolina-Chapel Hill, Chapel Hill, North Carolina, USA; and 5Boys Town National Research Hospital, Omaha, Nebraska, USA.

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and text of this article on the journal’s Web site (

The authors have no conflicts of interest to disclose. The authors had full editorial control of this work and manuscript.

Received April 29, 2015; accepted June 22, 2015.

Address for correspondence: Mary Pat Moeller, Boys Town National Research Hospital, 555 N. 30th Street, Omaha, NE 68131, USA. E-mail:

Back to Top | Article Outline


Longitudinal, multicenter studies are complex enterprises, yet represent an essential approach for addressing research gaps related to children who are hard of hearing (CHH) (Tomblin & Hebbeler 2007). (For a Table of Acronyms, see Supplemental Digital Content, Appendix A, In this section, we describe various features of the Outcomes of Children with Hearing Loss (OCHL) project that supported both the science and the collaborative process of implementing the study. This information is shared to support others who are working to establish collaborative science of a longitudinal nature. For each lesson in the following section, we introduce the potential issues or problems facing longitudinal, multisite studies, followed by the approaches that the OCHL team found to be successful in addressing these needs.

Back to Top | Article Outline


A common finding of the majority of past studies involving CHH is considerable between-subject variability in outcomes. In addition, before the implementation of newborn hearing screening, it was challenging to recruit samples of sufficient size to fully explore the complex, potentially interacting, child, family, and intervention factors that were influencing outcomes and explaining individual differences. There was a pressing need to recruit a sample of sufficient size to allow for multivariate modeling and analysis and to do so by incorporating multidisciplinary perspectives and a whole child view (Moeller et al. 2007; Tomblin & Hebbeler 2007). The adoption of an accelerated longitudinal design (Stanger & Verhulst 1995) served the project well in meeting this overarching need. If the team had relied on a traditional longitudinal design with all participants entering as infants and followed prospectively from that point, the sample sizes would have been insufficient to address the primary research questions using multivariate analyses. Furthermore, the adherence to a schedule of testing children around their birthdays allowed the team to derive both cross-sectional and longitudinal views of children’s outcomes as the study progressed. Another advantageous feature of this design was that sample sizes for any particular age grouping systematically increased as children “aged up” into the next assessment wave. Both features supported the team’s efforts in interpretation of the data and theory formation as the project evolved. This enrollment approach and related longitudinal waves of testing yielded a robust sample size that supported the use of multivariate methods and growth modeling (Tomblin et al. 2015b, this issue, pp. 14S–23S).

Multivariate methods allowed the team to identify interactions that are not revealed with univariate approaches. Multilevel modeling was effective in examining the longitudinal findings and resulted in identification of a particularly interesting interaction between average-aided residualized speech intelligibility index and language growth (Tomblin et al. 2015a, this issue, pp. 76S–91S). This analysis revealed a cumulative effect of the degree of relative hearing aid (HA) benefit (i.e., residualized speech intelligibility index) on language; children with greater aided benefit made gains in language relative to children with normal hearing (CNH) and those with least aided benefit declined in language development relative to CNH. This nuanced interpretation supported by the analytic method strengthens the interpretation of links between aided audibility and child language outcomes. Another unique finding was the interaction between age at HA fit and the duration of HA use. This analysis reveals that early HA use is linked to better language outcomes during much of the preschool period. Yet, it also reveals that later receipt of HAs does not result in irreversible language deficits and that as duration of HA use increased, systematic growth in language was observed. These key interactions are revisited to stress the value of multivariate methods as we seek to understand the interplay of variables influencing young children within and across developmental stages.

Back to Top | Article Outline


There are a number of common threats to validity that researchers work to minimize (Shadish et al. 2002). One example is a selection bias where there are pre-existing differences between enrolled participants and those who do not elect to participate or other differences that limit the generalizability of the findings to larger populations. The objective of the OCHL research team was to recruit a large, epidemiological sample of CHH and a matched sample of CNH. Although extensive efforts were made to recruit unbiased samples from a socioeconomic perspective, the final participants in both groups came from homes that are more advantaged than the typical US population. This is a common issue in longitudinal studies, given the barriers to ongoing participation for caregivers from lower socioeconomic backgrounds. However, it is an important issue to consider when interpreting this body of work. To address the primary research questions, it was necessary to accept participants from homes where at least one parent was speaking English in the home. Clearly this is an important control but it results in a sample that does not represent today’s global society.

Future studies should examine similar questions about CHH living in homes where caregivers are linguistically and culturally diverse. Some researchers have approached selection bias in innovative ways, and such efforts should be broadened and supported. For example, Hurtado et al. (2008) conduct research in a community-based laboratory in a low-income neighborhood. This laboratory includes bicultural/bilingual staff and it is a robust example of bringing the research to the community. The University of Iowa brings the research to the caregivers’ communities using a mobile testing van. Another example resides at Boys Town National Research Hospital, where staff in the Human Research Subjects Core (funded by the National Institute on Deafness and Other Communication Disorders) regularly work with cultural leaders in Hispanic, African-American, and Native American communities to increase awareness about and reduce barriers to participation in research. Because it appears to be especially difficult to recruit CHH from lower income homes, efforts to coordinate research with public education programs should be considered. By bringing the research to the schools, the costs to caregivers of participation in longitudinal studies could be minimized. Examples like the longstanding practice of state-wide assessments in Colorado (Yoshinaga-Itano et al. 1998) and the current efforts of the National Early Childhood Assessment Project (Yoshinaga-Itano 2015), a multistate project to collect state-wide outcomes data, are examples of efforts that yield samples that are representative of the typical US population. Attrition is another threat to validity in longitudinal studies, as participants drop out or are lost to follow up. In the OCHL study, strategic steps were taken to promote retention of participants (Tomblin et al. 2015b, this issue, pp. 14S–23S). Community-based professional collaborations were formed and maintained through periodic contacts, including newsletters with updates on progress and results. Efforts were made to maintain consistent contact with caregivers participating in the study. Testing sessions were followed by brief written reports regarding the child’s performance that could be shared with schools. Family interviews were scheduled at the 6-month point between annual visits to support the goal of staying connected with the research study. Family-friendly newsletters, child birthday greetings, and posting on social media sites were other forms of ensuring consistent contact and appreciation for participation. The final attrition rate over the 5-year study was minimized to an overall level of 9.27%.

In multisite studies, there can be systematic differences between sites that can compromise the integrity of data collection. For example, lack of oversight of test administration and scoring could result in procedural differences that could affect the validity and reliability of test scores. Quality controls were put in place from the inception of the study to further minimize threats to validity (Tomblin et al. 2015b, this issue, pp. 14S–23S). Multicenter studies require careful controls to ensure systematic implementation of procedures. An examiner with extensive experience in longitudinal outcomes studies provided training and ongoing supervisory feedback (through direct observation and review of videotaped sessions) to examiners at all three sites. Extensive procedural manuals were developed to support the training process. Procedural details, protocol revisions, and scoring details were carefully documented on a Sharepoint site that was accessible to all research team members. All test protocols were doubled scored by project examiners to minimize errors in deriving raw and standard scores.

Another challenge in multisite studies is the efficient management of data so that a collaborative analysis across sites is possible. To support efficiency in data management, an extensive and centralized database was developed and housed at the University of Iowa. This data repository allowed for efficient data entry from each of the sites through the Sharepoint site. All data were doubled entered into the program to minimize data entry errors, and the software for the data entry program automated the calculation of standard scores, which served as another verification of scoring accuracy. The database allowed for ease of data retrieval and contained tracking features that alerted the team to missing data or other issues that could be caught early in the process and addressed. These quality control features of the study benefitted from the experience gained and infrastructure built at the University of Iowa as a result of their longitudinal studies of children with specific language impairment (Tomblin et al. 1997).

Back to Top | Article Outline


Dr. Howard Gadlin (2007), National Institutes of Health Ombudsman, observed that “bringing together a talented group of people to work cooperatively to solve a problem takes time, commitment, passion, and a lot of hard work…and collaboration introduces into scientific work dimensions of interpersonal interaction that are not ordinarily considered very important in scientific work.” He noted the need for scientists to have skills in such areas as collaboration, interdependence, and joint problem solving, as well as the ability to develop collective responsibility as a team. In hindsight, these words offer sage wisdom, as the process of nurturing effective teamwork was instrumental to attaining the OCHL team goals. A number of resources are available from Responsible Conduct of Research sites that provide guidance from Gadlin and others about strategies to address the challenges of collaborative research (i.e.,

Strategic leadership and clear division of labor based on team members’ strengths were ingredients of successful collaboration. Leaders can play a key role in ensuring that multisite research studies are effective in reaching their goals and efficient in their overall functioning, and the OCHL project had co-principal investigators (Co-PIs) charged with this responsibility. Over time, those individuals discovered that they had complementary strengths, a circumstance that can be particularly advantageous in collaboration if recognized (Wagner & Muller 2009). The principal investigators discovered how to leverage their respective strengths through clear role assignment and division of labor, and they worked to build a clear, common mission. Similarly, as the team gained experience working together, the complementary strengths of individual team members as well as the research sites became evident. This supported the process of allowing the Co-PIs to delegate clear roles and responsibilities in line with individual and institutional strengths. This included identifying leadership at all levels of the project.

Another realization, which came about by willingness to ask as a team if our processes were working effectively (Lencioni 2004), was that video conference calls were becoming laden with detailed and inefficient discussions of specific issues. By critically analyzing the process, the team discovered the need to assign discipline-specific issues to subgroups of team members, who would then bring a recommendation to the full team. Disciplinary subgroups were formed under the topics of audiology, speech-language-pre-academics, service provision, and social-emotional development, and leaders were charged with oversight of the work of each subgroup. These subgroups often realigned in the process of manuscript preparation, because most studies involved multidisciplinary expertise. This process of cross-fertilization for data analysis and manuscript preparation was formative in working toward a single entity team, rather than separate cooperating institutions. Although that took some time to achieve and has not been without challenges, the collaborators became interdependent, working on common missions of the team. What evolved was a sense of collective responsibility and ability for team members to hold one another accountable for outcomes (Lencioni 2004). Administrative support in the form of a grants manager has been invaluable in tracking details, creating and monitoring shared budgets and resources, and enabling consistent and effective team communication.

Clear and frequent communication among team members has been a key ingredient to ensuring effective and efficient collaboration. Strategies implemented include (1) monthly interactive video conferences, (2) annual face-to-face meetings of the full team (2.5 days), (3) annual meetings of key investigators with consultants and advisory board members, (4) annual face-to-face meetings of principal investigators, and (5) regular subgroup phone calls. In addition, the Sharepoint site mentioned above provided a mechanism for raising issues and questions for team comment. Team members were held accountable for being inclusive in their communication. An aspect of communication that was vitally important was the development of guidelines regarding authorship decisions, internal manuscript review, and timing of research dissemination. In regard to authorship, the team adapted recommendations from the International Committee of Medical Journal Editors (which can be downloaded from to develop clear written guidelines for authorship. If conflicts arose, the guidelines were used to resolve them. In the final summation, investment in processes to support team collaboration was well worth the time and effort.

In summary, multisite research collaborations require careful attention to scientific issues and to processes that promote effective collaboration. The scientific efforts can be compromised without effective infrastructure and a shared commitment to work toward collective goals. Scientific efforts are fostered when time is taken to attend to processes related to team communication, accountability, and the highest quality implementation.

Back to Top | Article Outline


This research was supported by the following grants from the National Institute on Deafness and Other Communication Disorders/National Institutes of Health: R01DC009560, R01DC013591, and R03DC012647.

Back to Top | Article Outline


Gadlin H. Rethinking scientific teams: Competition, conflict and collaboration. (2007)Paper presented at the Nanotechnology and Society: The Organization and Policy of Innovation workshopAmherst, MA Retrieved October 30, 2014, from
Hurtado N., Marchman V. A., Fernald A.. Does input influence uptake? Links between maternal talk, processing speed and vocabulary size in Spanish-learning children. Dev Sci. (2008);11:F31–F39
Lencioni P. Death by Meeting. (2004) San Francisco, CA Jossey-Bass
Moeller M. P., Tomblin J. B., Yoshinaga-Itano C., et al. Current state of knowledge: Language and literacy of children with hearing impairment. Ear Hear. (2007);28:740–753
Shadish W. R., Cook T. D., Campbell D. T. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. (2002) Independence, KY Wadsworth Cengage Learning
Stanger C., Verhulst F. C.F. C. Verhulst, H. M. Koot. Accelerated longitudinal designs. In The Epidemiology of Child and Adolescent Psychopathology. (1995) New York, NY Oxford University Press:385–405
Tomblin J. B., Hebbeler K.. Current state of knowledge: Outcomes research in children with mild to severe hearing impairment-approaches and methodological considerations. Ear Hear. (2007);28:715–728
Tomblin J. B., Records N. L., Buckwalter P., et al. Prevalence of specific language impairment in kindergarten children. J Speech Lang Hear Res. (1997);40:1245–1260
Tomblin J. B., Harrison M., Ambrose S. E., et al. Language outcomes in young children with mild to severe hearing loss. Ear Hear. (2015a);36:76S–91S
Tomblin J. B., Walker E. A., McCreery R. W., et al. Outcomes of children with hearing loss: Data collection and methods. Ear Hear. (2015b);36:14S–23S
Wagner R., Muller G. Power of 2: How to Make the Most of Partnerships at Work and in Life. (2009) New York, NY Gallup Press
Yoshinaga-Itano C. Evolution of a Public Health Revolution: Universal Newborn Hearing Screening. (2015)Paper presented at the American Auditory Society ConferencePhoenix, AZ
Yoshinaga-Itano C., Sedey A. L., Coulter D. K., et al. Language of early- and later-identified children with hearing loss. Pediatrics. (1998);102:1161–1171

Managing attrition; Quality assurance; Research collaboration; Selection bias; Strategic leadership

Supplemental Digital Content

Back to Top | Article Outline
Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.