Hard to Define, Harder to Find Patients: Using AI & Real-World Data to Understand Treatment Resistant Depression

Mental health conditions can take a staggering toll on an individual physically, socially and financially. Many patients languish with partially effective or ineffective treatments, even though new options are constantly entering the market. At OM1, we are using our extensive mental health data network and AI technology to create a more precise way of identifying patients with treatment-resistant depression in order to break the cycle, broaden access to advanced therapies, and improve outcomes.

Treatment-resistant depression (TRD) typically refers to an inadequate response to two or more antidepressant trials of adequate dose and duration within a current depressive episode. TRD is relatively common in clinical practice, with 5% to 50% of the patients not achieving adequate response following antidepressant treatment. The financial burden is high, with estimates of TRD accounting for nearly half of the $92B annual cost of depression in the US.

Why are we talking about finding TRD patients?

Identifying patients with TRD in real-world data allows us to understand the full context of their health journey and work towards earlier identification, which leads to better outcomes. Insights can be gained into less common patient subtypes or underrepresented groups to help prevent patients from ‘falling through the cracks.’ The ultimate goal is to use data and technology to identify patients in need and direct them to and accelerate the time to appropriate treatments. This is especially important for those misdiagnosed with other conditions or those unaware of more advanced therapeutic options – when getting to the right treatment sooner can meaningfully and positively impact outcomes.

AI for patient identification in TRD: the ‘gold standard’ challenge

Artificial intelligence technologies have many uses, but in healthcare, they excel at identifying and then applying complex patterns in large datasets. Trained on very large-scale, real-world data, tools like OM1’s AI Patient Finder™ learn to distinguish signals associated with conditions of interest. We then use these signals to highlight patients to focus on in other datasets for research and in the clinic as an aid to medical decision making. To calibrate our AI, we needed a ‘gold standard’ set of known TRD patients – but while it is relatively easy to identify patients with major depressive disorder (MDD) in real-world data through diagnostic codes, there is no clearly established approach to identifying patients with TRD.

So how do we find them?

Below, we outline three approaches to defining a gold standard TRD patient within our MDD dataset of over 400,000 patients with clinical notes linked to medical and pharmacy claims.

Option 1: Use the regulatory definition of two failed trials of different antidepressants of adequate dose and duration within a depressive episode. This definition seems simple enough, and is familiar to those working in TRD, but the devil is in the details. In particular, there’s no agreement on ‘adequate.’ This makes identifying a consistent set of TRD-positive patients challenging. It is also sometimes hard to define an ‘episode’ of depression, so this definition invariably misses patients who haven’t had a change in medication recently, possibly excluding those most in need of other therapeutic options.

Option 2: Use ‘no argument’ TRD qualifiers – specifically, a history of somatic therapies used to treat TRD such as electroconvulsive therapy, transcranial magnetic stimulation, or vagal nerve stimulation (and alternatively, use of IV or inhaled ketamine). While patients with a history of these treatments are in almost every case inarguably treatment-resistant, using this option misses anyone who hasn’t made it this far in their treatment journey or who has simply fallen through the cracks. Given that these therapies are relatively uncommon, often expensive, and difficult to access for many, a definition based on them is inevitably incomplete.

Option 3: A final option is to define a TRD cohort by clinician attestation – using advanced text extraction techniques applied to the unstructured clinical notes written by psychiatrists who document that patients with an established diagnosis of major depressive disorder are experiencing treatment-resistance.

We chose Option 3 because we have found that unstructured data are often very useful when a condition is commonly seen and documented by clinicians, but not coded, like TRD. This approach is akin to ‘polling’ hundreds of experts to find cases and using them as a training cohort. OM1 used this approach to calibrate our AI to identify patterns of TRD in our depression dataset. We found over 3,000 patients with electronic health records with verifiable psychiatrist-attested TRD.  

How well did the model perform?

After we calibrated Patient Finder with patient records that met the gold standard of documentation of TRD by their psychiatrist, we evaluated performance using the standard area under the receiver operating curve (AUC) metric that reflects how well the model balances sensitivity and specificity. A score of 0.5 is no better than a coin flip, and a score of 1.0 represents a perfect model. Our model exhibited very strong performance with an AUC of 0.87, suggesting robust ability to distinguish psychiatrist-labeled TRD cases from other MDD patients.

Can we clinically validate the model?

When it comes to mental health and depression, the PHQ-9 is one of the most commonly reported outcomes measures used in clinical care and in clinical trials. For example, in the Annals of Internal Medicine publication Harmonized Outcome Measures for Use in Depression Patient Registries and Clinical Practice, PHQ-9 scores are espoused as a standard to understand a patient’s depression severity. We used our extensive PHQ-9 scores to validate our new AI model of TRD.

Our hypothesis going into this analysis was that patients with worse disease activity are more likely to be treatment-resistant. For example, using our physician-defined TRD group, we saw more TRD representation when depression severity was worse. We also found a similar pattern using patients identified by the AI Patient Finder tool: patients the model identified as more likely to be TRD had worse PHQ-9 scores. This is good confirmation that we’re finding the right people.

The final step is to prove the value of the model in supporting faster patient identification and better treatment access in the clinic. We are doing this by working with health networks to integrate our technology and data into the clinical workflow in a way that is useful and actionable for healthcare professionals with the goal of benefiting patients.

Summary

Depression is a common and highly debilitating disorder that needs more than good therapies to address it. We need a coordinated effort that includes more screening, education on how to make our brains more resilient and operate at their peak in a world where we are overwhelmed with news, information and technology every day. OM1’s AI Patient Finder for TRD  uses both structured and unstructured portions of the EMR to identify patients in need. We used a unique approach to define a “gold standard” that draws on the judgment of treating clinicians, and then calibrated our AI to find TRD  patients using a vast amount of structured data going back years. We look forward to applying this technology to help identify more TRD patients and help to improve their outcomes.

Learn more about OM1’s work in TRD, mental health, or AI by contacting Maria Demko (mdemko@om1.com), Kim Waldron (kwaldron@om1.com) or visiting our website.