When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.1
In this quote, Lord Kelvin argues for the importance of objective measurement. This concept has become a fundamental principle of management that applies in the commercial, nonprofit, and even academic domains.
Within academia, publication tracking provides one objective measure of scientific quality and productivity.2,3 It also allows institutional leaders to evaluate the overall scientific portfolio of a specific program or the institution as a whole. Metrics such as publication frequency, total number of articles, average number of citations per article, citation rates, number of publications in top percentiles, measures of interdisciplinary nature and specialization, number of publications with particular key words, types of journals, impact factor of journals, number of citations for combinations of funding sources, multi-institutional coauthorship, and other characteristics can be captured or calculated and linked with sources of support and funding.4 Within information science, the study of bibliometrics and author disambiguation can produce sophisticated, quantitative citation or content analysis of journal articles.5–8
All of these metrics are particularly useful for evaluation purposes when institutions are faced with resources that they must equitably distribute to the entire research community, such as those provided by the National Institutes of Health (NIH) through the Clinical and Translational Science Award (CTSA) program.9–13 Publication tracking provides institutional leaders with quantifiable metrics to ensure the prudent stewardship of federal, foundation, and institutional investments by allowing them to identify the greatest scientific return for various types of support. Institutions often resort to ad hoc surveys tied to grant reporting cycles to identify the publications that arose from particular funding sources.14 However, this method may not be exhaustive, and some publications or pertinent information may be missed because of imprecise recall by publication authors.
Searches in NIH databases like the Research Portfolio Online Reporting Tools (RePORTER)15 and PubMed16 can provide coarse details concerning publications and supporting grants. However, data from these systems indicate only that a publication was supported by a grant, not how and at what capacity the publication was supported. Gathering that granular information is necessary for managing large-scale service-oriented portfolios. Thus, in 2010, we developed a straightforward, yet dynamic methodology for identifying and attributing recently published journal articles with a known pool of researchers receiving support from the CTSA program.
In this article, we describe the methodological framework and workflow that allows researchers to attribute a journal article to specific institutional resources at or near the time of publication. Additionally, we outline the measures of impact used in this new automated workflow for citing publications over the first 30 months of its operation. Finally, we present a set of practical lessons learned to inform other institutions or programs seeking to employ a systems-based approach for resource tracking and the evaluation of scientific portfolios.
A Pilot Funding Model in Vanderbilt University’s CTSA Program
For a grant application to NIH or to another federal or nonprofit agency to receive serious consideration, the applicants generally must provide substantial preliminary data.17 Yet, funding for pilot studies to generate such data is often slow, and sometimes unavailable. The Vanderbilt Institute for Clinical and Translational Research (VICTR) was created in 2006 in part to stimulate new research ideas and to provide a home for researchers who want to plan and conduct scientific studies.18 To streamline an important component of translational science—moving swiftly from hypotheses to proof of concept and then to full-scale investigation—as part of VICTR, we created an openly accessible, tiered model to support pilot funding initiatives. This model allows researchers to apply at any time for any level of research support for pilot funding; for research infrastructure services needed to generate preliminary data; or to support projects that are small and do not fit the scope of the traditional “R” award process.
Since October 2007, more than 1,700 pilot and supplemental awards for hypothesis-driven research have been granted to clinical and translational investigators. VICTR pilot funding is available to all faculty members, schools, departments, and disciplines across Vanderbilt University School of Medicine and Meharry Medical College (931 researchers from 217 departments and disciplines).
The VICTR Portfolio Management System
All VICTR pilot funding requests are created and submitted online through a single portal, StarBRITE.19 The application process starts with a “preliminary information” section—the principal investigator provides coarse detail concerning the nature of the support she is requesting (e.g., letter of support for a grant submission, resources to be funded by VICTR, sponsor-reimbursed use of VICTR resource infrastructure) and a summary of her study. On the basis of the information entered in this first section, the system algorithmically refines the options that appear in the “pick resources” section that comes next, providing a list of resources, from a pool of over 250 items, tailored to the researcher’s specifications. Researchers can select any items from the list; each item includes instructions for selection (e.g., quantity, rationale) as well as the associated cost or value. Selected items appear in a “shopping cart,” which displays aggregate cost/value and can be reviewed by the research team.
The total amount of VICTR funding requested determines the review path and the level of scientific justification required. Vouchers—VICTR funding requests that total less than $2,000—require minimal scientific justification and are typically reviewed administratively within 48 hours (with retrospective full scientific oversight). Expediated requests that total between $2,000 and $10,000 require a five-page NIH-style scientific proposal, relevant ancillary budget documents, and researcher biosketch and are pre-reviewed by VICTR staff and reviewed by at least one VICTR Scientific Review Committee member. Requests exceeding $10,000 require documentation similar to expediated requests but always require full Scientific Review Committee review and deliberation. In addition, research teams are required to specify the expected outcomes (e.g., publication, pilot data for grant submission) and other information important for assessing the portfolio (e.g., research type and area, similar to NIH reporting list, phase of study).
The principal investigator finalizes the request and verifies that she will cite the VICTR funding in any subsequent publications associated with the resources requested.
Once the pilot funding application is submitted, the StarBRITE system facilitates a scientific review and awarding process. When an application is approved, StarBRITE provides resource-specific instructions for the researcher to redeem the award. Automated data feeds from institutional accounting systems inform the StarBRITE system when the award has been redeemed.
In addition to the pilot funding request and approval mechanism, VICTR also provides researchers with a number of tools and services (e.g., REDCap20 and ResearchMatch21) that they can request and use immediately without going through the pilot funding request and approval process. In these cases, StarBRITE records and maintains a comprehensive list of direct-to-researcher services used by individual projects and attributes their use to the principal investigators who lead these supported projects.
The closed-loop data and workflow methods embedded in the StarBRITE system support VICTR administrators in managing and tracking resources. Thus, the administrators know what resources have been requested, provided, and used by what research teams. However, because future research outcomes (e.g., publications) are disconnected from this request and reward process, evaluating the relative impact of VICTR-resource distribution requires additional information capture and linkage methods leveraging external databases.
Designing a Systematic Approach to Automated Publication Matching
Figure 1 illustrates the StarBRITE workflow, designed to detect new publications, notify researchers, and identify when a publication acknowledges VICTR support. Fundamental components of our publication-matching methodology include (1) a “listening” service that detects new publications in PubMed that have been authored by VICTR researchers, (2) a multifaceted notification process for contacting VICTR researchers when new publications are detected, (3) a straightforward adjudication process enabling researchers to match detected publications with specific VICTR services, and (4) program-wide evaluation and management dashboards generated from researcher-provided publication and resource attribution data. The notification and adjudication system is dynamic and easily accommodates new VICTR services as they are developed and offered to research teams. The listening service and notification process allow researchers to attribute resources within days of their publication appearing in PubMed. Key system components are described in more detail below.
Publication detection and researcher matching
In StarBRITE, we created an automated mechanism to query PubMed nightly to identify any publications affiliated with Vanderbilt or Meharry that had been added to the database in the last six months. The six-month window ensures that we did not miss potential publications because of the order in which they were added to or curated in PubMed. Our method for computationally interrogating PubMed leverages the application programming interface technology supplied by the National Library of Medicine.22
For each new Vanderbilt or Meharry publication that StarBRITE detects, we pull the list of authors and cross-reference the names on that list with our known pool of VICTR-supported researchers. To do so, we use a researcher-to-resource mapping table that StarBRITE autogenerates when VICTR services are initially requested, awarded, and used. If any author has used one or more VICTR-supported resources, a notation is made in an underlying database table identifying the publication as a potential match for VICTR resources.
In StarBRITE, we designed four methods for requesting publication/resource adjudication from researchers whenever new publications of interest are detected and logged. First, a congratulatory e-mail is sent to the researcher listed as an author with instructions for the next steps for adjudication. Second, the researcher is presented with an “on until cleared” notification dialog (a visible, highlighted action-required reminder) each time she uses the StarBRITE system for any purpose. Third, the StarBRITE system has embedded mechanisms preventing researchers from requesting new or amending existing VICTR support until all previously identified publication/resource match candidates have been adjudicated. The final notification mechanism is tied to “anniversary reporting” feedback that is required from all researchers receiving VICTR support. With this mechanism, researchers are directed to a prepopulated StarBRITE Web form, requesting a progress report and outcomes to date for the project. Previously adjudicated publication information is included in this anniversary report; the report also affords researchers the opportunity to supply information on publications not previously detected by the StarBRITE system (e.g., multicenter publications for which the corresponding author is at another institution).
The initial step in the publication/resource-matching process is author disambiguation. After receiving notification of a newly detected publication (e.g., e-mail or StarBRITE “on until cleared” notification dialog), researchers are presented with two choices: (1) “Yes, I am the author of this publication”; or (2) “No, I am not the author of this publication.” If the researcher selects the “No” option, the StarBRITE system logs the author and publication match as false. If the researcher selects the “Yes” option, she is redirected to a short form in the StarBRITE system that displays a list of all the VICTR-supported resources that she has been awarded and used in the past with instructions that she should indicate which resources contributed to that publication. As the final step in the adjudication process, researchers are asked to cite the CTSA grant in PubMed.16
Storing adjudicated data, evaluation, and management reporting
After publication/resource adjudication is complete, the matched data are stored in StarBRITE and added to the appropriate dashboards to support management decisions in evaluating the scientific return on investment for individual projects and overall VICTR services.
Assessment of Response to and Effectiveness of Publication Matching
Figure 2 provides a cumulative count of the known VICTR-supported publications and their publication dates for the time period between October 2007 and February 2013. It took several months before researchers were aware of VICTR and the support available, then additional time for publications to evolve from a supported study to a published article. By February 2008, however, we were able to attribute approximately 8 publications per month to VICTR-supported projects.
Between October 2007 and February 2010, researchers attributed their publications to VICTR services as part of a yearly request for information coinciding with the required CTSA annual progress report. A new system was launched in September 2010. It recognizes and queues for immediate adjudication any publication authored by a known VICTR researcher that has been published since the February 2010 annual data collection survey. More important, the new system allows for the daily detection and adjudication of publications, instead of the annual process. We found a marked change in the number of publications supported by one or more VICTR services—by the end of our reporting period, approximately 22 publications per month were attributed to VICTR services.
A quantitative summary of the publication-matching, adjudication, and attribution processes between September 2010 and February 2013 is shown in Figure 3. Using a loose (first initial and last name) matching algorithm, the StarBRITE system notified every known VICTR-supported researcher each time a new PubMed publication was detected. Roughly 77% (6,412/8,365) of all potential researcher/publication matches were adjudicated, with 10% (831/8,365) attributing one or more VICTR service to the publication. A majority of the researchers who responded to a matching request did so because of an e-mail notification (81%); fewer responded because of an “on until cleared” StarBRITE notification (14%), the mandatory requirement to adjudicate all known publications before requesting new resources (3%), or as part of the anniversary reporting process (1%). Of adjudicated potential researcher/publication matches, 16% (1,003/6,412) were determined to not be a match, highlighting the need for a low-burden adjudication workflow. We have not analyzed thoroughly the 23% (1,953/8,365) of potential matches for which a researcher was contacted but never responded. Some of these cases likely include junior investigators (e.g., trainees, fellows) who received VICTR support and authored a publication before leaving the university. The StarBRITE system does not maintain a forwarding e-mail address, and access is automatically revoked once faculty and staff have left the university.
Although general attribution of VICTR services was sufficient to fulfill the annual reporting requirements of our funding organization, we sought more granular information to assist VICTR management decisions. Table 1 shows data generated by the StarBRITE system and used by VICTR leadership to assess the uptake and value added by selected services. Some services and tools were available during the entire reporting period (e.g., pilot program from 2007 to 2012), whereas others were added in later years (e.g., ResearchMatch in 2010). The data presented in Table 1 are automatically extracted as part of the researcher adjudication process and displayed in evaluation dashboards within the StarBRITE system.19 Other dashboards automatically created from data captured during the publication attribution adjudication process include data regarding the scientific focus areas supported and co-citations with NIH agencies. Figure 4 provides a snapshot of the scientific focus areas supported by VICTR. These and other dashboards inform and assist institutional leaders in assessing the impact of the entire portfolio of VICTR-supported services and tools.
In this section, we describe our experience building and operating the StarBRITE portfolio management system and present advice that may be helpful to other institutions that want to deploy similar tracking and real-time evaluation methods.
Cast a wide net in gathering researcher input
We had initial concerns that contacting multiple researchers (all detected publication authors with known VICTR support) for input concerning a single publication might be overly burdensome with questionable value added. In practice, we found that contacting multiple authors of the same publication for adjudication improved the accuracy of the attribution process. In complex multidisciplinary scientific projects, authors’ responsibilities can be diverse during the planning, conduct, and scientific writing stages, and individual team members may be unaware of all VICTR services used.
Anticipate author disambiguation issues
Given our “wide net” approach, based on a loose matching algorithm (e.g., first initial and last name), we were concerned about receiving complaints from researchers who were mistakenly contacted because of author disambiguation. In practice, disambiguation issues were limited to a handful of cases. For example, in one case, we found two VICTR-supported researchers who shared a similar first initial and last name pairing. One was a prolific author, and the other was a research staff member with few known or expected publications. To eliminate future erroneous requests for the staff member, we built into the system an opt-out flag enabling the VICTR portfolio management team to designate individual VICTR-supported researchers to be ignored during the matching and adjudication process. In another case, the StarBRITE system was not detecting publications for a VICTR-supported researcher because the system knew him by his middle name rather than his first name. We remedied this and similar detection issues by adding a PubMed author string field to the VICTR researcher database. When populated, the system searches detected publications for the value in this field rather than the automatically generated first initial and last name normally used.
Build multiple trigger points
VICTR promotes a culture of service to the research enterprise and regularly requests feedback from this community to develop innovative solutions based on unmet needs. Research teams realize that evaluation is a key program component, understand the value of VICTR support, and recognize that that support relies on grant funding with progress report requirements. We have found that researchers will help with publication attribution if the process is not burdensome. In particular, congratulating researchers on the new publication and then asking for their help is an effective process. Only a minority of researchers ignored this adjudication request. In these cases, not allowing future VICTR support requests until adjudication is complete works well. Finally, adding a separate mechanism for researchers to include publications not detected by the system (e.g., a multicenter publication for which the corresponding author is at another institution) is necessary to capture all relevant publications.
Anticipate program evolution
Matching publications to VICTR-supported resources requires system-level data related to the use of each resource. Thus, we built into our notification and adjudication system the ability to easily create data feeds related to new VICTR services as they are added to the program. Since implementation of the VICTR program, eight new resources have become available and were easily incorporated into the StarBRITE system.
Consider a “just-in-time” workflow
Prior to designing and implementing the just-in-time publication adjudication process that we described in this article, we used a semiautomated matching process designed to help VICTR-supported researchers report their publications. As part of our old process, once a year, we asked researchers to provide information about all known VICTR-supported publications to fulfill annual reporting requirements for our CTSA grant. This process was similar to the anniversary reporting we described earlier, except that all reporting was done in February to fulfill reporting requirements for a single CTSA grant progress report. The old process included the automated retrieval of information from PubMed to assist with publication attribution. Although confounding factors exist in our new system (e.g., a wider pool of researchers/staff contacted and services added to VICTR and the adjudication process over time), we feel that a major contributor to the increased rate of attribution is the switch to a just-in-time process rather than collecting information annually.
Consider the potential for program management
Our current system has the ability to draw broader conclusions related to our overall scientific portfolio and resource contributions. Simple dashboards (e.g., the data presented in Table 1 and Figure 4) are effective for real-time assessment and provide a deeper understanding of resource use and a global view of the overall scientific portfolio. Combining resource use information with resource cost data enables VICTR leadership to make more informed management decisions. Comparing resources awarded and publication history at the researcher level can help to assess future individual project support requests.
Conclusions and Next Steps
VICTR’s mission is to transform the way ideas and discoveries make their way back and forth across the translational research spectrum. The program accomplishes this goal in part by providing pilot funding and support to a large number of diverse research teams. Active portfolio management and evaluation are essential elements for creating and successfully running a large infrastructure program, like VICTR. Since implementing the just-in-time researcher notification and service-level adjudication system, we have seen measureable improvements in our ability to determine the scientific return on investment for VICTR-supported projects.
Building an automated system requires planning, initial investment, and maintenance. Creating our pilot project service request and fulfillment system in StarBRITE required a high degree of planning and approximately nine months of developer effort.19 Other VICTR services, such as REDCap and ResearchMatch, were developed independently over time.20,21 Because we started from a point where we could easily use StarBRITE to query and collate data on the use of all services, the publication attribution workflow we described here was relatively inexpensive to implement (approximately four weeks of programmer effort). Anticipating the future growth of CTSA services, we built a highly dynamic data architecture and can establish data feeds from new CTSA services that we want to include in the publication adjudication process using minimal resources (e.g., 10 programming hours or less per new service).
Before we implemented our current portfolio management and publication adjudication system, the process of gathering and reporting information to NIH required many dedicated weeks from VICTR support staff, numerous information-gathering surveys directed to VICTR researchers, and intensive manual review. Since implementing the StarBRITE portfolio management system, resource tracking and annual reporting requirements are nearly completely automated and require minimal effort by VICTR support staff.
We will continue to grow our system to accommodate new VICTR services as they are added. We are exploring additional ways of creating information dashboards centered around individual researchers and academic departments to further enhance the ability of the VICTR Scientific Review Committee to make informed decisions when considering new large project resource requests. We also are considering methods for combining data obtained from the publication adjudication process to create just-in-time StarBRITE search functionality designed to “find experts” at Vanderbilt and Meharry (e.g., to find a mentor, a collaborator, or a VICTR Studio expert). Finally, we are creating and evaluating the StarBRITE publication awareness tools to ensure that researchers comply with NIH publication access policies.
1. Thomson W. Electrical units of measurement. Popular Lectures and Addresses. 2011;Vol 1 Cambridge Library Collection–Physical Sciences. Cambridge, England Cambridge University Press
2. Agarwal P, Searls DB. Can literature analysis identify innovation drivers in drug discovery? Nat Rev Drug Discov. 2009;8:865–878
3. Kissin I. Can a bibliometric indicator predict the success of an analgesic? Scientometrics. 2011;86:785–795
5. De Bellis N Bibliometrics and Citation Analysis: From the Science Citation Index to Cybermetrics. 2009 Lanham, Md Scarecrow Press
6. Gurney T, Horlings E, van den Besselaar P. Author disambiguation using multi-aspect similarity indicators. Scientometrics. 2012;91:435–449
7. Aronson AR, Lang FM. An overview of MetaMap: Historical perspective and recent advances. J Am Med Inform Assoc. 2010;17:229–236
8. Kang IS, Na SH, Lee S, et al. On co-authorship for author disambiguation. Inf Process Manag. 2009;45:84–97
9. DiLaura R, Turisco F, McGrew C, Reel S, Glaser J, Crowley WF Jr.. Use of informatics and information technologies in the clinical research enterprise within US academic medical centers: Progress and challenges from 2005 to 2007. J Investig Med. 2008;56:770–779
10. Druss BG, Marcus SC. Tracking publication outcomes of National Institutes of Health grants. Am J Med. 2005;118:658–663
11. Hall KL, Stokols D, Stipelman BA, et al. Assessing the value of team science: A study comparing center- and investigator-initiated grants. Am J Prev Med. 2012;42:157–163
12. Patel VM, Ashrafian H, Ahmed K, et al. How has healthcare research performance been assessed? A systematic review. J R Soc Med. 2011;104:251–261
13. Patel VM, Ashrafian H, Almoudaris A, et al. Measuring academic performance for healthcare researchers with the H index: Which search tool should be used? Med Princ Pract. 2013;22:178–183
14. Ovseiko PV, Oancea A, Buchan AM. Assessing research impact in academic clinical medicine: A study using Research Excellence Framework pilot impact indicators. BMC Health Serv Res. 2012;12:478
17. Cech TR. Fostering innovation and discovery in biomedical research. JAMA. 2005;294:1390–1393
18. Kain K. Promoting translational research at Vanderbilt University’s CTSA institute. Dis Model Mech. 2008;1:202–204
19. Harris PA, Swafford JA, Edwards TL, et al. StarBRITE: The Vanderbilt University Biomedical Research Integration, Translation and Education portal. J Biomed Inform. 2011;44:655–662
20. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–381
21. Harris PA, Scott KW, Lebo L, Hassan N, Lightner C, Pulley J. ResearchMatch: A national registry to recruit volunteers for clinical research. Acad Med. 2012;87:66–73
23. Pulley J, Clayton E, Bernard GR, Roden DM, Masys DR. Principles of human subjects protections applied in an opt-out, de-identified biobank. Clin Transl Sci. 2010;3:42–48
24. Danciu I, Cowan JD, Basford M, et al. Secondary use of clinical data: The Vanderbilt approach. J Biomed Inform. 2014;52:28–35
© 2015 by the Association of American Medical Colleges
25. Byrne DW, Biaggioni I, Bernard GR, et al. Clinical and translational research studios: A multidisciplinary internal support program. Acad Med. 2012;87:1052–1059