Research to Policy: Innovative Bibliometrics Reveal How CTSA Publications Advance Translation
The National Institutes of Health’s Clinical and Translational Science Awards (CTSA) program aims to accelerate translational research by providing support to investigators in order to bring new knowledge from the bench to the bedside via changes in public awareness, practice guidelines, health policy, and legislation. Nevertheless, it can be difficult to measure the impact of such support on translational outcomes, such as in public policy changes.
In a new study published in Academic Medicine, lead author Nicole Llewellyn, PhD, Emory University School of Medicine, Associate Director, Georgia CTSA Evaluation and Continuous Improvement, evaluated how CTSA-supported research has extended its impact beyond academia to influence public policy. Using cutting-edge bibliometric tools, Georgia CTSA partnered with CTSA evaluation leaders at Case Western Reserve University and Virginia Commonwealth University to demonstrate how CTSA-supported research has been used in public health and policy literature.
“This study is one of the first to evaluate translational advancement from CTSA-supported scientific research to public policy. Very little is known about how health science research is used outside of academic silos; however, this is necessary for successful translation,” says Dr. Llewellyn.
Bibliometrics, or the analysis of publication data, is a well-established method for evaluating a program’s impact. Traditional bibliometric approaches to evaluating programs include assessment of supported publications and grants and accrued academic citations. Though much is known about how translational research is disseminated through peer-reviewed publications, less is known about the transfer of knowledge outside of academia.
“Our goal in this study was to better understand how CTSA-supported research influences public health policy. New analytic tools have allowed us to paint a broad picture of how this works, as well as to draw direct, documented lines from CTSA support, to research publications, to policy literature,” shares Llewellyn.
The evaluation was conducted using 3 complementary approaches, aiming to: (1) summarize overall policy influence of the CTSA consortium; (2) provide illustrative case examples of CTSA-supported research translated to real-world health policy impact; and (3) determine the types of research that are most likely to influence policy. Using PubMed, 118,490 publications were identified as receiving support from a CTSA hub from the inception of the program in 2006 through 2021. Articles were queried in the Overton policy database, which indexes references to publications in global policy literature. The search revealed 13% of CTSA-supported articles were referenced in policy documents, significantly more than the expected proportion (10%) calculated by Overton for similar articles. References came from 576 policy source outlets across 87 countries, predominantly the United States and Europe. The most frequent sources included guidelines in PubMed Central, the World Health Organization, and the Centers for Disease Control and Prevention.
“Cross-hub collaborations related to evaluation are critical to apply rigorous evaluation methods, which help to better illustrate the impact and reach within the CTSA program,” states co-author Deborah DiazGranados, PhD, Evaluation Director and Associate Professor, Virginia Commonwealth University School of Medicine.
“Similarly, the publications resulting from CTSA hub collaborations show significant interdisciplinary overlap, something that has been shown to have longer lasting influence and more impact on the fields covered by clinical and translational science,” adds co-author Clara Pelfrey, PhD, Evaluation Director and Associate Professor, Case Western Reserve University School of Medicine.
The authors also illustrated policy impact by describing 6 research articles based on CTSA-supported research, including their bibliometric information, the number and type of policy documents that cited those articles, and the public health purpose of select policy documents. For example, the Georgia Clinical and Translational Science Alliance (Georgia CTSA) Pediatrics program provided clinical nursing support to research in an article analyzing maternal influenza immunization and reduced likelihood of prematurity in births, published in PLOS Medicine in 2011. As of January 2022, this article had accumulated 31 policy references. One of those sources included a report by the World Health Organization, Safety of immunization during pregnancy: a review of the evidence: Global Advisory Committee on Vaccine Safety, where it was cited as evidence of the benefits of vaccines.
Lastly, different descriptive and inferential analyses were used to show how articles referenced in policy literature differ from those that have not. The evaluation found that articles with greater clinical relevance, Altmetric Attention (i.e., nonacademic community/public attention), and academic citation influence were more likely to be referenced in policy literature. These findings help to characterize the kinds of research that have influenced and may be expected to influence health policy in the future.
In demonstrating how supported research has stretched beyond the bounds of academia to influence policy, Dr. DiazGranados concludes, “This research has been valuable to not only see the impact of the science supported by the CTSA consortium but also to refine how we define impact when evaluating the CTSA program.”
The paper, Translating Scientific Discovery Into Health Policy Impact: Innovative Bibliometrics Bridge Translational Research Publications to Policy Literature, was published online on April 10, 2023, in Academic Medicine.