Can We Trust the Data? The Case for Validation

Real world research has produced recent newsworthy findings on topics as diverse as COVID, mental health, cancer, and dementia. But much of the published RWE, even among studies that end up in headlines, doesn’t include crucial information readers need to know in order to trust the results—by that I mean information on the validity of the link between the coded information that forms the basis of the study and the actual health states of interest.

A recent study in JAMA reported a 5-fold increase in suicide-related ED visits in adolescents.  But what the authors actually found was an increase in the use of a single ICD-10-CM code. And in addition to suicidal ideation, that code is used for low self-esteem, excessive crying, and other symptoms.

I’m not suggesting there isn’t a huge mental health crisis underway in this country, and I wouldn’t doubt that more young people are contemplating suicide. However, the authors did not provide, nor could I find in their references, any evidence of validation of the link between this ICD code and actual suicidal ideation. That is, of all the people with the code, how many truly had thoughts of suicide? And conversely, of all the people with suicidal thoughts, how many were diagnosed with this code?

A classic study from more than a decade ago found that only about 1 in 20 RWE studies provided any evidence for the validity of the codes used, leading the authors to conclude, “People with a code frequently do not have the condition it represents.”

I ran my own simple version of this study: I examined the 10 most recent RWE studies available on PubMed that used ICD coding. Seven^1-7 didn’t mention the validity of their codes or algorithms. Of the three remaining, one⁸ was primarily a report of the validation of a new algorithm (sensitivity of 58% and specificity of 85%), another⁹ conducted a simple validation as part of the study (sensitivity 25–‍35%, specificity 69–90%), and the last used codes that had been validated in a prior study (sensitivity 90%, specificity 83%)¹⁰.

This is an improvement from a decade ago—3 in 10 is more than 1 in 20—but the quality of the codes still varied considerably.

So the JAMA paper isn’t an outlier, but I’m certain things will be different in 10 years. The demand for validated patient reported outcome measures skyrocketed once there was a clear path to including them in drug labels. PROs appeared infrequently in labels at the turn of the millennium, about 20% from 2005–2015, and more than 25% in the last few years. Labeling claims based on RWE are on the early part of a similar curve.

Over the last 10 years, my colleagues at PHAR and I have conducted validation research that runs the gamut from simple (e.g., pull 10 medical records and confirm the codes for a single condition) to complex (e.g., multi-center studies at academic institutions aiming to validate algorithms for subtle clinical differences between patients^11-19).

Some RWE researchers don’t want to be bothered doing this back-to-basics work, and it’s often incorrectly seen as unnecessary. But the validation train is getting ready to leave the station—get on board now or prepare to be left behind!

Dr. Michael Broder, a board-certified obstetrician and gynecologist, has 30 years’ experience in health economic and outcomes research. He received his research training in the Robert Wood Johnson Clinical Scholars Program at UCLA and RAND, attended medical school at Case Western Reserve University, and received his undergraduate degree from Harvard University.

In 2004, Dr. Broder founded PHAR, a clinically-focused health economics and outcomes research consultancy.  PHAR is a team of dedicated, highly trained researchers —individuals who are singularly focused on delivering high-quality health economics and outcomes research insights to the life science industry. PHAR has successfully conducted hundreds of studies resulting in more than 800 publications on a wide variety of therapeutic areas, and maintains an expansive network of collaborators, including 8 of the top 10 academic institutions in the US, as measured by NIH funding. Download our bibliography here.

Unencumbered by corporate bureaucracy, PHAR can efficiently execute contracts and complete projects on time and on budget. PHAR prides itself on being reliable and responsive to clients’ changing needs, and welcoming the challenge of tackling problems others can’t.

1. Im PK, Wright N, Yang L, Chan KH, Chen Y, Guo Y, et al. Alcohol consumption and risks of more than 200 diseases in Chinese men. Nat Med. 2023 Jun;29(6):1476-86.
2. Isath A, Malik AH, Goel A, Gupta R, Shrivastav R, Bandyopadhyay D. Nationwide analysis of the outcomes and mortality of hospitalized COVID-19 patients. Curr Probl Cardiol. 2023 Feb;48(2):101440.
3. Sultan P, Guo N, Kawai M, Barwick FH, Carvalho B, Mackey S, et al. Prevalence and predictors for postpartum sleep disorders: a nationwide analysis. J Matern Fetal Neonatal Med. 2023 Dec;36(1):2170749.
4. Nguyen TN, Qureshi MM, Klein P, Yamagami H, Mikulik R, Czlonkowska A, et al. Global impact of the COVID-19 pandemic on stroke volumes and cerebrovascular events: a 1-year follow-up. Neurology. 2023 Jan 24;100(4):e408-21.
5. Bommersbach TJ, McKean AJ, Olfson M, Rhee TG. National trends in mental health-related emergency department visits among youth, 2011-2020. JAMA. 2023 May 2;329(17):1469-77.
6. Lee JH, Park HJ, Kim S, Kim YJ, Kim HC. Epidemiology and comorbidities in idiopathic pulmonary fibrosis: a nationwide cohort study. BMC Pulm Med. 2023 Feb 4;23(1):54.
7. Barer Y, Ribalov R, Yaari A, Maor R, Arow Q, Logan J, et al. Healthcare utilization, costs, and epidemiology of Huntington’s disease in Israel. Clin Park Relat Disord. 2023 Jun 28;9:100208.
8. Daluwatte C, Dvaretskaya M, Ekhtiari S, Hayat P, Montmerle M, Mathur S, et al. Development of an algorithm for finding pertussis episodes in a population-based electronic health record database. Hum Vaccin Immunother. 2023 Dec 31;19(1):2209455
9. Mellhammar L, Wollter E, Dahlberg J, Donovan B, Olséen CJ, Wiking PO, et al. Estimating sepsis incidence using administrative data and clinical medical record review. JAMA Netw Open. 2023 Aug 1;6(8):e2331168.
10. Orandi BJ, McLeod MC, MacLennan PA, Lee WM, Fontana RJ, Karvellas CJ, et al. Association of FDA mandate limiting acetaminophen (paracetamol) in prescription combination opioid products and subsequent hospitalizations and acute liver failure. JAMA. 2023 Mar 7;329(9):735-44.
11. Cherepanov D, Raimundo K, Chang E, Eagan M, Zazzali JL, Solari PG, et al. Validation of an ICD-9-based claims algorithm for identifying patients with chronic idiopathic/spontaneous urticaria. Ann Allergy Asthma Immunol. 2015 May;114(5):393-8.
12. Exuzides A, Yermilov I, Campos C, Gibbs SN, Broder MS, Cohan S, et al. Using cognitive interviews to develop a conceptual claims-based algorithm to identify patients with neuromyelitis optica spectrum disorder. Poster presented at: 37^th Congress of the European Committee for Treatment and Research in Multiple Sclerosis; 2021 Oct 13-15; Virtual Meeting.
13. Exuzides A, Yermilov I, Dalglish H, Gibbs SN, Broder MS, Cohan S, et al. Testing a claims-based algorithm to identify patients with neuromyelitis optica spectrum disorder. Poster presented at: Annual Meeting of the Academy of Managed Care Pharmacy (AMCP); 2022 Mar 29-Apr 1; Chicago, IL.
14. Exuzides A, Yermilov I, Dalglish H, Gibbs SN, Broder MS, Cohan S, et al. Testing a claims-based algorithm to identify patients with neuromyelitis optica spectrum disorder. Poster presented at: 7^th Annual Forum of the Americas Committee for Treatment & Research in Multiple Sclerosis (ACTRIMS); 2022 Feb 24-26; West Palm Beach, FL.
15. Exuzides A, Yermilov I, Dalglish H, Gibbs SN, Reddy SR, Chang E, et al. Development and validation of a claims-based algorithm to identify patients with neuromyelitis optica spectrum disorder. Poster presented at: 38^th Congress of the European Committee for Treatment and Research in Multiple Sclerosis (ECTRIMS); 2022 Oct 26-28; Amsterdam, the Netherlands.
16. Greene N, Gibbs SN, Broder MS, Farnett L, Chang E, Campos C, et al. Identifying adult patients with nonrelapsing secondary progressive multiple sclerosis using algorithms in US-based healthcare databases Poster presented at: Academy of Managed Care Pharmacy (AMCP) Nexus; 2023 Oct 16-19; Orlando, FL.
17. Greene N, Gibbs SN, Broder MS, Farnett L, Chang E, Campos C, et al. Identifying adult patients with non-relapsing secondary progressive multiple sclerosis using algorithms in US-based healthcare databases. Poster presented at: ISPOR Annual International Conference; 2023 May 8-10; Boston, MA.
18. Khairnar R, Mahajerin A, Shapiro A, Sidonio R, Okolo A, Decker-Palmer M, et al. Claims-based algorithms to identify persons with severe hemophilia A. Poster presented at: ISPOR Annual International Conference; 2022 May 16-18; National Harbor, MD.
19. Raimundo K, DeCotiis B, Hussain I, Rehman S, Tilles S, Shahab N, et al. Validation of an ICD-9-Based claims algorithm for identifying patients with chronic idiopathic/spontaneous urticaria. Poster presented at: 19^th ISPOR Annual International Meeting; 2014 May 31-Jun 4; Montreal, Canada.