Getting the most from clinical experts in the time of COVID-19: Lessons learned
April 5, 2021
Gathering experts and soliciting their advice is a time-honored tactic. But methods for doing so invariably involve meeting in person.
Although the threat of COVID-19 appears to be slowly receding, business travel may take years to return to pre-crisis levels. In this climate, how can pharma researchers most effectively leverage advisors?
I have been conducting expert panels since the mid-1990s when, working as a faculty member at the UCLA School of Medicine and a researcher at the RAND Corporation, I was introduced to a technique for developing consensus among experts called the “RAND/UCLA modified Delphi panel” method. Over the last 25 years, in more than 40 publications, I have used this method to guide clinical experts to develop practice guidelines, disease classification systems, research agendas, quality improvement interventions, and more.
Until early last year, I moderated all of these meetings in person, but the pandemic forced us to adapt. Starting in March 2020, PHAR began conducting these panels using a virtual meeting format, and we have successfully done so multiple times since then.
By making a few key changes, we have used virtual meetings to lower the cost and shorten the timeline needed for a modified Delphi panel, without any appreciable negative impact.
Anyone who’s ever gotten 10 experts together knows that managing the discussion can be a challenge. Plans can quickly go awry when one or two vocal participants ignore the rules of “playing nice.” Add to that the distractions inherent in a video meeting and problems can multiply. But that’s when the Delphi method shows its true strength.
Researchers at RAND developed the Delphi method in the 1950s as a way to eliminate “specious persuasion,” where the person with the strongest convictions or greatest supposed authority pushes others to agree against their own judgment. The method used successive rounds of surveys, with expert respondents shown the group medians after each round and asked to explain their reasoning if their answer was outside a certain range. Several rounds were conducted in this way until “consensus” emerged.
In the 1980s several changes were introduced, and the process was collectively called the “modified Delphi method”. These included:
- providing a formal summary of relevant literature to participants
- conducting an in-person discussion with a professional moderator
- using two (rather than indefinite) rounds of ratings
- defining “consensus” with a formal mathematical rule applied to the final ratings.
As in the original Delphi method, experts were never required to reach consensus during the meeting. Instead, the second-round ratings were used to summarize areas of agreement.
At PHAR, despite 16 years’ experience with these methods, we were anxious about transitioning to a virtual setting. Would panelists be able to focus on the task at hand? Would the level of discussion be curtailed? Could consensus still be achieved? Would panelists like it? (This last question was particularly important, as we have heard panelists enjoy these meetings and their enthusiasm helps make these panels a success).
Over the last 12 months, we’ve conducted 5 virtual panels with more than 60 expert panelists, and here’s what we’ve learned:
- Participants don’t miss the travel.
- Funders appreciate the reduced cost.
- Panel work can be intense, and multiple short remote sessions work better than do fewer, longer ones. We’ve moved from all-day or multi-day in-person meetings to multiple blocks of 2-4 hours over consecutive days, which lets physician panelists see patients on meeting days.
- International panels are harder. Travel may be difficult, but getting up at 2 AM to attend a panel meeting online is no fun.
- We miss the informal interchange among panelists that happened during breaks or meals.
- On the other hand, without this informal discussion, it is easier to keep the meeting on track and on time.
- The method survives. Across these panels, we have successfully reduced disagreement and achieved consensus on the topics of interest. Upcoming publications will discuss some of these findings in detail.
Developing clinical practice guidelines, disease classification systems, and diagnostic criteria are just some of the times when expert input is crucial to pharmaceutical company research. The modified Delphi panel method remains the best studied and most methodologically rigorous way to help health experts reach agreement on complex, clinical topics. Moving from in-person to virtual meetings is not without challenges, but there have been unexpected advantages. We anticipate conducting future panels using both methods, picking and choosing depending on the circumstances.
Contact Michael at mbroder@pharllc.com or visit pharllc.com.
About the author
Dr. Michael Broder, a board-certified obstetrician and gynecologist, has 30 years’ experience in health economic and outcomes research. He received his research training in the Robert Wood Johnson Clinical Scholars Program at UCLA and RAND, attended medical school at Case Western Reserve University, and received his undergraduate degree from Harvard University.
In 2004, Dr. Broder founded PHAR, a clinically-focused health economics and outcomes research consultancy. PHAR is a team of dedicated, highly trained researchers —individuals who are singularly focused on delivering high-quality health economics and outcomes research insights to the life science industry. PHAR has successfully conducted hundreds of studies resulting in more than 800 publications on a wide variety of therapeutic areas, and maintains an expansive network of collaborators, including 8 of the top 10 academic institutions in the US, as measured by NIH funding. Download our bibliography here.
Unencumbered by corporate bureaucracy, PHAR can efficiently execute contracts and complete projects on time and on budget. PHAR prides itself on being reliable and responsive to clients’ changing needs, and welcoming the challenge of tackling problems others can’t.
