"Inferential Challenges for Real-world Evidence in the Era of Routinely Collected Health Data: Many Researchers, Many More Hypotheses, a Single Database" (Panagiotou OA, Heller R, JAMA Oncology)

The Study

This paper was a published as JAMA Viewpoint that presented the statistical challenges of using real-world data (e.g. administrative claims data) for answering clinical and policy questions. We focused on examining the implications of selective inference (formulating research hypotheses that are then examined with the same data) and multiple testing (pursuing numerous research questions within the same database). We describe how these issues, if unaccounted for, can lead to biased results including increased number of false-positive associations. We also propose some statistical techniques to overcome these challenges.

Was there Anything About Your Research Findings that Surprised You? 

In our experience, we found it very concerning that multiplicity at the community level is rarely accounted for; i.e. almost never are statistical analyses adjusted for what other researchers have done using the same database. We were also surprised by how widespread some selective inference practices are, e.g. researchers performing many different analyses until they found a significant result.

How Did You Become Interested in this Work? 

The lack of replicability in science is what got me interested in this; to a large extent, this lack of replicability is due selective inference and multiple testing.

How Does Your Research Relate to Your Advance-CTR Big Data Award?

I, in collaboration with Dr. Tom Trikalinos, received an Advance-CTR Big Data Pilot award in 2017 for our study, "Computational Heath Services Research to Identify Use of Low-Value Care in Rhode Island." 

We looked at administrative claims data from the RI All-Payer Claims Database (APCD) to understand why Rhode Island patients receive some of the highest rates of low-care in the nation. Using machine learning techniques, we developed a predictive algorithm to understand what factors lead to the utilization of low-value care in our state.

This project builds on our Big Data award by looking at administrative claims databases and electronic medical records, and asking why they are prone to bias. We found that because there is no pre-registration of studies and researchers have easy access to the databases, this allows investigators to analyze the data in myriads of ways. 

How Does Your Research Affect Health and/or Healthcare in Rhode Island?

Our findings can inform the analyses of RI-based databases such as the RI APCD, thereby leading to improved and unbiased inferences (e.g. reduced numbers of false-positive findings). 

Read the Paper