Abstract
Real world data (RWD) and their sources are increasingly available and used to generate clinical evidence, called real-world evidence (RWE). They offer new methodologies to study patient experiences and outcomes in previously unstudied, real-world settings. However, it is unclear what kinds of clinical evidence these data can reasonably be expected to generate, and how the devices that produce these data should be studied and regulated. Accordingly, the aim of this thesis was to investigate how RWD – from personal digital device data to large EHR and claims databases – may replace or augment current clinical research methods, and to highlight how personal digital device clinical studies and regulatory pathways should prioritize evaluating patient experience, outcomes, and safety.
The shortcomings of randomized control trials (RCTs), including expense, time, and lack of generalizability, have driven growing interest in alternative research approaches utilizing RWD. Despite this enthusiasm, it remains unclear whether it is reasonable to expect that observational research methods can truly replace RCTs, or whether they will augment and complement the clinical evidence generated in RCTs.
Personal digital devices offer the opportunity to study new, real-world settings that have not previously been included in traditional research methods. In particular, post-procedural recovery is an outside-hospital setting in which associations between physical activity, patient reported outcomes, and clinical outcomes are not well understood.Currently, traditional medical device regulatory pathways are used to evaluate digital health interventions, and limited clinical evidence is generated for these products. In 2017, the U.S. Food and Drug Administration (FDA) announced a new Precertification Program for Software as a Medical Device, which will evaluate and authorize manufacturers, rather than evaluate individual products. This new program is currently in a pilot phase, and it remains unclear what quantity and quality of clinical studies will be required of participants in the program.
To address these questions, we conducted 3 distinct yet related research studies. One study, evaluating the feasibility of using RWD to replicate clinical trial evidence, found that only 15% of clinical trials published in high-impact journals in 2017 could be replicated using currently available RWD sources. A second study of post-procedural recovery found meaningful associations between patient activity and irregular heart rhythms as recorded by a mobile ECG device. The results of the third study characterizing clinical studies of the digital health products manufactured by companies in FDA’s Pre-Cert Pilot suggest that the available clinical evidence for digital health interventions is usually insufficient to rigorously evaluate a product’s efficacy.
While RWD and personal digital devices offer exciting new ways to generate clinical evidence, the opportunities and limitations of these methodologies must be communicated to clinicians, researchers, and the public. There is a growing role for observational data to complement, but not completely replace, RCT data. More rigorous studies of personal digital device data are needed to ensure that these novel technologies aid clinical decision-making and advance patient care. Lastly, digital health regulatory pathways, such as the Precertification Program, must explicitly require stronger and more generalizable evidence to ensure patient safety and well-being.
