In the progressive landscape where artificial intelligence (AI) converges with healthcare, particularly in precision oncology, researchers at Flatiron Health have achieved a significant breakthrough. Led by Aaron B. Cohen, MD, head of research oncology and clinical data, the team has demonstrated how large language models (LLMs) can revolutionize cancer treatment by extracting critical data from unstructured electronic health records (EHRs). This development marks a pivotal moment in harnessing AI’s potential to improve patient outcomes through more accurate and analyzable healthcare data.
The Quest for Accurate EHR Data Extraction
Aaron B. Cohen’s interest in clinical decision-making emerged during his medical oncology training when he observed that many patients were receiving aggressive treatments towards the end of life, prompting him to understand the reasons behind these decisions. His experiences fueled his ambition to align treatments more closely with patients’ goals, improving the quality of end-of-life care.
After completing his training, Cohen joined Flatiron Health, drawn by the company’s vast network of patient records containing rich clinical data. Flatiron Health’s goal of leveraging this data to support improved clinical decision-making and patient care resonated deeply with him. By continuing to treat patients weekly at Bellevue in New York City, he gained firsthand experience with the complexities of EHR documentation and the essential role accurate details play in clinical trials, decision support, and research. Cohen’s work set the stage for exploring how AI could bridge the gap in extracting valuable insights from these unstructured data sources.
Traditional Data Extraction Challenges
For nearly a decade, Flatiron Health relied on human abstraction, a method where trained individuals manually parse through free-text notes to extract relevant information and structure it for analysis. The painstaking process, while effective, raised a critical question for Cohen: could AI match the accuracy of human abstraction but on a larger scale?
The landscape of EHR documentation is riddled with inconsistencies and varied formats, making the precise automation of data extraction a desirable yet elusive goal. Though successful, the manual process was labor-intensive and resource-consuming, often lagging in scalability and efficiency. The quest for a more refined and consistent method of extracting data resulted from the need to overcome these bottlenecks and push the boundaries of what AI could achieve in the realm of healthcare data.
Experimenting with LLMs
To address these challenges, Cohen and his team embarked on a comparative study to evaluate the efficacy of large language models (LLMs) versus traditional deep learning models in extracting clinical data from EHRs. Traditional deep learning models have a proven track record of identifying patterns and relationships within structured data, excelling in specific metrics and tasks. However, the core advantage of LLMs lies in their ability to understand and generate human language, allowing them to process less structured data by predicting word sequences.
The team centered their experiment on PD-L1 biomarker data, given its critical role in cancer treatment. PD-L1 levels directly influence treatment decisions and are relevant across various cancer types. Cohen’s team evaluated both zero-shot LLMs, used in their native, unmodified state, and fine-tuned LLMs, which were adapted to perform specific tasks such as outputting results in JSON format for easier analysis. The goal was to ascertain whether these models could effectively and accurately extract vital information from EHRs.
Fine-tuning for Accuracy
The results of the study revealed stark differences between the capabilities of zero-shot and fine-tuned LLMs. While zero-shot LLMs exhibited some potential in extracting PD-L1 data, they often produced inaccuracies and hallucinations, limiting their reliability. On the other hand, fine-tuned LLMs demonstrated a marked improvement in accuracy, delivering consistently precise results despite the variability in cancer types, documentation practices, and timelines.
Remarkably, the fine-tuned LLMs outperformed traditional deep learning models even with fewer training examples. This finding underscores the adaptability and potential of LLMs in processing and extracting valuable information from complex and varied data sources. For Cohen and his team, this success not only validated the feasibility of using AI for EHR data extraction but also opened the door to broader applications within Flatiron Health’s extensive clinical network.
Broad Applications of LLMs
The implications of these findings extend far beyond data extraction, suggesting a range of applications within Flatiron Health’s operations. One promising area is clinical trial matching, an essential process that requires comprehensive knowledge of relevant trials and criteria. The proficiency of LLMs in understanding and processing human language makes them well-suited to this task, potentially enhancing the efficiency and accuracy of matching patients to appropriate clinical trials.
Another priority for Cohen is enhancing clinical decision support, particularly in differentiating responses to immunotherapy based on PD-L1 levels. By accurately and consistently extracting data, LLMs can provide oncologists with critical insights that inform treatment decisions, facilitating more individualized and effective cancer treatments. These applications exemplify the transformative potential of LLMs in supporting targeted, evidence-based approaches to patient care.
Challenges and Future Directions
Despite the promise shown by LLMs, Cohen is cautious about viewing them as standalone solutions for clinical decision support. The multifaceted nature of required data and prediction algorithms poses significant challenges that cannot be fully addressed by LLMs alone. However, by making underlying data more accessible, LLMs can still play a crucial role in aiding AI and machine learning (ML) integration into high-stakes clinical decisions.
The complexity of biomarker results, such as EGFR mutations, highlights the need for detailed, layered information extraction. LLMs have the potential to streamline this process, allowing oncologists to focus on critical findings without getting bogged down by the need to manually sift through extensive data. As cancer treatments continue to evolve, the ability to efficiently and accurately extract and analyze complex biomarker data will be increasingly important in guiding personalized treatment strategies.
Educational Implications
Cohen emphasizes that the trustworthiness of LLMs and similar AI tools is contingent upon the users’ trust in the data they provide. This reliance underscores the critical importance of computational literacy among clinicians, who need to understand how LLMs function, recognize their limitations, and maximize their utility. Effective use of AI in clinical settings requires a synergetic approach, blending advanced AI capabilities with the expertise and judgment of healthcare professionals.
To this end, Cohen advocates for integrating data-driven, computational approaches into medical education. He believes that equipping clinicians with the knowledge and skills to leverage AI tools effectively will enhance their ability to utilize these technologies to their full potential. By fostering a deeper understanding of AI among medical professionals, the healthcare industry can better harness the power of these tools to improve patient care and outcomes.
Potential Impact on Medical Education
In the progressive landscape where artificial intelligence (AI) intersects with healthcare, particularly in precision oncology, researchers at Flatiron Health have made a significant breakthrough. Under the leadership of Aaron B. Cohen, MD, head of research oncology and clinical data, the team has illustrated how large language models (LLMs) can transform cancer treatment by extracting vital information from unstructured electronic health records (EHRs). This advancement is a major milestone in leveraging AI’s potential to enhance patient outcomes through more precise and analyzable healthcare data.
By using LLMs, researchers can sift through the vast, often chaotic, EHRs to find crucial details that might otherwise be overlooked. These AI models can interpret complex medical jargon, identify patterns, and pull out relevant patient information, making it easier for oncologists to understand each patient’s unique medical history. This leads to more personalized treatment plans, thereby increasing the chances of successful outcomes.
Furthermore, this breakthrough has broader implications beyond just oncology. The methods developed by Cohen’s team could be applied to other areas of medicine, thereby improving the overall efficiency and effectiveness of healthcare systems. This innovative use of AI in the medical field heralds a new era where data-driven decisions can lead to better treatments and ultimately, better patient care.
