This episode examines a systematic benchmark of six commercial large language models applied to plasma cell-free RNA across three clinical cohorts, assessing LLM-driven gene-panel nomination and autonomous classifier construction versus conventional statistical workflows.
0:00Welcome to Base by Base, a paper cast that brings genomics to you wherever you are. Thanks for listening, and don't forget to follow and rate us in your podcast app. You know, usually when we think about developing a single diagnostic biomarker, uh, like a biological red flag that tells a doctor exactly what disease you have.
0:16Well, it's a monumental grueling effort. Oh, absolutely. can take years of lab work. Right. years of work, 1000000s of dollars. And it requires researchers to essentially dig through infinite molecular haystacks, running these complex statistical models, just hoping to find a viable signal.
0:33And the reality of that process is brutal. I mean, less than one% of published biomarkers ever actually get FDA approval. Less than one percent. That is wild. It's a massive bottleneck in modern medicine.
0:44The human body produces just vast amounts of data, right? And distilling all that transcriptomic noise down to a reliable clinical test requires profound computational and biological expertise. So this brings up a pretty wild idea for this deep dive.
1:00What if we could just hand that entire molecular haystack to a large language model and say, you know, find the needle, write the code, and build me a diagnostic test? It sounds a bit like science fiction.
1:11Right. Like, how could this change the speed at which we diagnose complex, mysterious diseases? I mean, we use these AIs to synthesize text or fix syntax errors in a script? Routine tasks mostly. Exactly.
1:24But asking them to autonomously write code that diagnoses a human being from raw, messy biological data. That feels like a massive leap. I'm highly skeptical that a language model can act as a fully functional bioinformation.
1:36Well, that skepticism is entirely warranted, and it's exactly the boundary being tested in the research we're analyzing today. We're looking at a direct test of whether artificial intelligence can transition from simple language generation into autonomous scientific discovery.
1:51Before we get into the mechanisms of how they actually tested this, we need to acknowledge the architects of the study. Today we celebrate the work of Hunter Gaudio, Andrew Bliss, and the team at the Mindig School of Biomedical Engineering at Cornell University, who have advanced our understanding of integrating AI into diagnostic biomarker discovery.
2:11So, to evaluate what that team achieved, we 1st need to define the specific haystack they are feeding into these AI models. In this deep dive, the biological data in question is plasma, cell free, RNA, or CFRNA.
2:26Okay, CFRNA. Let's make sure we ground this in the biology for a second. When cells in the body die or undergo severe stress, they don't just vanish. Right, if they burst. Right. They burst and release fragments of their RNA into the bloodstream.
2:38So looking at CFRNA is essentially like analyzing the shredded documents floating in the blood to figure out exactly what tissues are injured. Yeah, or what specific proteins those cells were trying to manufacture right before they died.
2:50And that's a highly accurate way to visualize it. It gives us a minimally invasive, real-time transcriptomic snapshot of the entire body. But because it's a mixture of fragments from multiple organ systems, it is incredibly noisy data.
3:05Like a giant puzzle with a bunch of missing pieces. Exactly. So to test if an AI could actually filter through that noise and find a true diagnostic signal, the researcher set up a calibration curve of biological knowledge using 3 distinct clinical cohorts.
3:21They didn't just give the AI one target, they created a gradient of difficulty. Precisely. The 1st cohort focused on Kawasaki disease, or KD versus MISC. And those are both hyper-inflammatory conditions in kid.
3:34Yeah, too closely related conditions. They share very similar overlapping clinical presentations like high fevers, rashes, systemic inflammation. But MISC is specifically a post-COVID syndrome. Exactly.
3:45Differentiating them at the molecular level is challenging, but absolutely critical for directing the right immunosuppressive therapies. Ah, okay. So you're establishing a baseline. Two known, relatively well documented conditions with a localized inflammatory response.
3:59Right. Then the 2nd cohort scales up to a massive global health challenge, which is tuberculosis or TB, compared to symptomatic respiratory controls. And TB represents a vast diagnostic gap worldwide. A huge gap.
4:14Millions of people go undiagnosed annually because traditional sputum-based tests require the patient to produce a high quality sample. Which is often really difficult, especially for children. Exactly, or for extra pulmonary cases.
4:27So a reliable, blood-based biomarker for TV is kind of a holy grail in global health. And because TB is one of the oldest and most studied infectious diseases on the planet, there's just a massive footprint of data in the scientific literature.
4:40Mountains of it. Which brings us to the 3rd cohort, the ultimate challenge, right? MECFS, myalgic encephalomyelitis, or chronic fatigue syndrome. Compared to healthy sedentary controls, yes. An MECFS is defined entirely by clinical symptoms, like post-exertional malaise.
4:57With 0 validated molecular biomarkers. Absolutely zero. It remains a complete biological mystery. Which makes this a brilliant experimental design. If you want to know what an AI is actually capable of, you can't just test it on TV, where the answers are essentially already in the back of the book.
5:11Exactly. By using this gradient, they test the limits of the AI's knowledge synthesis. Can it only regurgitate the consensus on TV or can it genuinely discover novel biology in a mystery like MECFS? That is the core question.
5:26To find out, they pitted 6 state of the art models against each other, right? I did. They used open AIs, 03 and GPT 40, Anthropics Claude Opus 4 and 3.7 Sonnet, and Google's Gemini 2.5 Pro and Gemini 2.0 Flash.
5:40And they evaluated them across 2 highly complex tasks. Task one was feature selection. Okay, so they provided the AI with a prompt and a list of genes found in the CFRNA data. Yeah, asking it to select the top 200 most predictive genes for a diagnostic panel.
5:55So just picking the best 200 variables out of 1000s of shredded transcriptomic fragments. seems straightforward enough. Well, task 2 is where the rigor dramatically increases. It was end-to-end automation.
6:06They gave the AIs, the raw CFRNA count matrices. TNO giant spreadsheets of numerical biological data. Exactly. And instructed them to build a machine learning classifier entirely from scratch, train it on the data, and predict the diagnoses on a hidden test data set.
6:24Wait, let's pause and appreciate what that entails. We aren't asking a chat bot to write a summary paragraph. We are asking it to operate as a senior data scientist. That's fully autonymously. It has to ingest a numerical matrix, write a Python script, choose an algorithm like a random forest, handle data normalization, perform cross validation to tune the hyper parameters, and then execute performance reporting.
6:47And if the matrix dimensions don't align, It has to read its own traceback error in the terminal and rewrite its own function to fix the bug. That is wild. That self-correction loop is the defining feature of autonomous coding agents.
6:59It is. However, you'd expect these advanced models to execute this flawlessly, but the reality was quite different. Prompt adherence was a significant hurdle right out of the gate in the 1st task. Right, here's where it gets really interesting.
7:11The hallucinations here reveal a profound limitation in how these models operate. GPT 40 and Gemini 2.0 flash hallucinated wildly. They really did. When asked to pick 200 genes from the provided human CFRNA list.
7:26They just ignored the instructions and started returning protein aliases, retroviruses, and entirely non-human genes. Which totally exposes the danger of using a language model for raw biological science without rigorous guardrails.
7:40Because at its core, an LLM is a token predictor, not a biological oracle. It sees the word gene, and calculates the statistical probability of the next sequence of letters based on its training data. Exactly.
7:53If a viral gene was mentioned frequently next to a human inflammatory marker and some random paper at red, it just spits out the viral gene. Completely oblivious to the fact it's trying to diagnose a human being.
8:03Yeah, that fundamental lack of biological grounding is why generative AI requires immense benchmarking in a medical context. It is worth noting, however, that the newer reasoning models demonstrated significant progress.
8:14All right, open AIO3, Gemini 2.5 Pro, and Anthropics Claude 3.7 Sonnet, followed the strict parsing instructions incredibly well. They did. So filtering out the models that failed basic instructions. Let's look at the models that actually built the 200 gene panels.
8:29To know if the AI found the right biological signals, the researchers had to compare the AI's selections against a gold standard. So they compare the LOM panels against 2 baselines. The 1st was a completely random selection of genes just to ensure the AI was doing better than chance.
8:45Right. And the 2nd was the traditional gold standard in transcriptomics, differential gene expression, or DGE. Let's quickly define DGE because it's crucial to understand what the AI is competing against.
8:56Differential gene expression isn't a machine learning algorithm, it's rigorous classical statistics. It's pure math Right. A bioinformatician uses mathematical models to calculate exactly which genes are significantly upregulated or down regulated between the sick patients and the healthy controls.
9:14Applying strict P value corrections to rule out false positives. Exactly. And when we look at the 1st cohort, KD versus MISE, that pure math one. It did. The LLM panels easily beat random guessing, proving they were extracting real biological signal, but they could not beat the predictive performance of the traditional DGE statistics.
9:33But the biology the AI uncovered was still highly relevant, right? For KD and MISC, the AI is accurately zeroed in on canonical immune pathways. picking genes heavily involved in IL 6 and NFKB signaling.
9:47Yes, these are massive, well-known inflammatory cascades. So the AI didn't beat the math, but it proved it understood the biological assignment. It knew it was looking at an inflammatory disease and picked the genes that drive inflammation.
9:59A solid demonstration of knowledge retrieval, but the tuberculosis cohort is where the results become truly disruptive. For TB, the AI panel is matched, and in several configurations actually exceeded the predictive performance of the human guided DGE statistics.
10:13Wow. And it did this by leveraging specific historical knowledge. Open AIO3 and Gemini 2.5 Pro, independently selected a gene called GBP 5. Yes they did. I've seen that specific gene in the literature.
10:25That is a highly validated, well-documented biomarker for active tuberculosis, driven by the body's interferon response to the pathogen. Exactly. The AI didn't just stumble onto GBP 5. It read the historical consensus of TV research, recognized its importance, and applied it to the raw CFRNA data perfectly.
10:44It synthesized decades of global infectious disease research to optimize its selection. That's incredible. It is. But then we arrive at the final cohort, MECFS. And for this disease, the AIs completely fail to beat the standard DGE statistics.
11:00They only managed to achieve marginal gains over random guessing, struggling to find any cohesive transcriptomic signal. Which perfectly illustrates the limitation of the technology. The AI is essentially a mirror reflecting the current state of human knowledge on the internet.
11:14Tuberculosis has mountains of published research for the AI to train on, so it excels. MECFS is a neglected disease with virtually no molecular consensus in the literature. The AI cannot synthesize what isn't there.
11:27If the underlying mechanism of the disease isn't published, the language model cannot magically derive it just from looking at gene names. The language models are incredible synthesizers, but they are not engines of novo biological discovery without statistical grounding.
11:42Exactly. That brings us to task two, the end to end automation. Where we ask the AI to act as that full-fledged bioinformatician. Did they manage to write the Python code, tune the models, and predict the diagnoses without human intervention?
11:56Well, for most of them, the pipeline just collapsed. Gemini 2.5 Pro frequently tyingd out or through terminal errors and couldn't finish the execution consistently. The complexity of managing the matrix dimensions, handling the data scaling, and optimizing the hyper parameters was just too much.
12:11Ultimately, only open AIO3 and Claude Opus 4 could actually finish the job from start to finish. And how did their automated classifiers perform on the hidden clinical test data? For the KD versus MISC cohort, open AIO3 autonomously build a classifier that achieved an 86.7% accuracy rate.
12:30That essentially matched the traditional human built statistical pipeline. It executed the entire data science workflow with the same level of accuracy as a human expert. But that success didn't translate across the board.
12:42In the TB cohort, open AI 03 underperformed, hitting 76.3% accuracy compared to the human benchmark of 82.6%. And for MECFS, it failed to improve upon the baseline at all, basically stagnating around a 57% accuracy rate.
12:59Claude Opus 4 performed significantly worse across all 3 cohorts. Well, it didn't sweep the board. The fact that an AI could autonomously construct a functional diagnostic pipeline is a paradigm shift.
13:11But there was a specific experiment within this task that yielded a highly counterintuitive result. The researchers tested the AIs using 2 different types of prompts, disease naive and disease informed.
13:22Meaning they either completely hid the clinical context and just handed the AI the numbers, or they explicitly told the AI, hey, you are looking at CFR and A data for tuberculosis. Exactly. When they use the disease informed prompts, you would naturally expect the predictive accuracy to increase.
13:36Right, because it has more context. But it didn't. Statistically, providing the medical context resulted in absolutely no significant improvement in the model's performance. Hold on. to push back on that.
13:47How is that mathematically possible? If I tell an advanced reasoning AI to look for tuberculosis. It has terabytes of TB pathology, immune response pathways, and transcriptomic papers embedded in its neural network weights.
14:02Oh, definitely. You're telling me it just ignored all that medical context when building the classifier. It didn't necessarily ignore it, but it couldn't utilize it at that stage of the pipeline. Once the AI is tasked with writing a machine learning classifier, like a random forest in psychit learn, the process becomes a pure mathematical optimization problem.
14:21Ah, I see. The text prompt doesn't alter the underlying logic of the algorithm fitting a curve to the data. The AI succeeded or failed in building these classifiers purely based on its fundamental python coding and data science skills, not its medical domain knowledge.
14:37It was acting as a software engineer, executing a script, not as a diagnostic physician evaluating a patient. That makes perfect sense. The biological knowledge is useful for selecting the genes in task one.
14:48But once you move to task 2 and start optimizing loss functions in code, knowing its TB doesn't change how a matrix multiplies. So synthesizing all of this. The token hallucinations, the massive win using historical knowledge and TV feature selection, and the harsh realities of end to end classification, what are the tangible implications for the future of clinical diagnostics?
15:10The primary takeaway is that while language models are surprisingly adept at extracting biologically meaningful signals, traditional human guided statistical workflows still deliver the highest predictive performance overall.
15:22They still win. They do. Furthermore, this benchmarking highlighted a very specific dangerous limitation in how LLMs execute these pipelines. When the AIs ran the end-to-end automation, they generally only outputted discrete class predictions.
15:36Let's explore why that is a massive problem in medicine. A discrete prediction means the AI simply outputs a binary text response, class one or class zero. Yes, the patient has TV or no, the patient does not.
15:47Right. But in clinical diagnostics, a binary yes or no hides the math. A doctor needs a probability distribution. They need to know if the model is 99.9% confident in that diagnosis or if it is hovering at like 51% confidence.
16:02Precisely. Because the AI failed to naturally provide those nuanced probability distributions, the researchers were severely restricted from calculating deeper, critical performance metrics, such as the area under the receiver operating characteristic curve or ROCAUC.
16:19Without ROCAUC, you cannot evaluate the trade-off between false positives and false negatives. If a TV test has a high false positive rate, you end up putting healthy people on aggressive toxic antibiotics for months.
16:31And if it has a high false negative rate, you send a highly contagious patient back into the community. You cannot build a safe medical test without that probability spread. And compounding that issue is the persistent looming threat of data leakage.
16:43These language models are pre-trained on a vast opaque scrape of the internet. It is incredibly difficult to definitively prove that the AI didn't just memorize a specific CFRNA data set or a related transcriptomic research paper during its massive training phase, which would give it an unfair artificial advantage in these diagnostic benchmarks.
17:03It's the equivalent of taking a rigorous open book exam when you've already memorized the professor's answer key. Yeah. You can't be 100% certain the AI is actually reasoning through the biology rather than just recalling a matrix it saw in 2024.
17:16Exactly. So if letting the AI run the entire laboratory pipeline end to end is fundamentally unsafe right now, what is the actual path forward for this technology in genomics? The most viable path forward is a highly integrated hybrid solution.
17:32We must stop viewing this as AI versus traditional statistics and start designing workflows that combine their respective strengths. Use traditional rigorous statistics like differential gene expression to handle the pure math.
17:42Yes. The stats filter out the noise, ensure mathematical accuracy and calculate the vital P values. Then, we deploy the language models to synthesize the vast mountains of medical literature. Prioritizing the most biologically plausible and actionable biomarker candidates from that statistically filtered list.
18:00Exactly. Let the statistics handle the mathematical optimization, and let the artificial intelligence read the library. That preserves the interpretability and safety of classical data science, while massively accelerating the biological discovery phase.
18:14It fundamentally reshapes the bioinformatic workflow. Large language models can effectively synthesize existing biomedical knowledge and autonomously execute diagnostic code, serving as powerful accelerators for biomarker discovery and well-steady diseases like tuberculosis.
18:31However, they still struggle with novel conditions, indicating that hybrid approaches combining human led statistical filtering with AI curation will be the most effective safe path forward. Without a doubt.
18:42Which leaves us with a profound concept to consider. We are moving from an era where the bottleneck was finding the biological signal to an era where the challenge is learning how to properly constrain and query in AI to validate that signal.
18:56What does this mean for the future of personalized medicine? When a hybrid system might soon be able to synthesize the entire history of human biology, and write a custom, statistically validated diagnostic test from a single drop of your blood in just 5 minutes.
19:10It's fascinating thought. This episode was based on an open access article under the CCBY4.0 license. You can find a direct link to the paper and the license in our episode description. If you enjoyed this, follow or subscribe in your podcast app, and leave a 5 star rating.
19:26If you'd like to support our work, use the donation link in the description. Now stay with us for an original track created especially for this episode and inspired by the article you've just heard about.
19:35Thanks for listening, and join us next time as we explore more science, base by base.
