When New Information Conflicts with What AI Knows
Johns Hopkins study reveals how large language models fail to ignore information learned during training.
By Dino Lencioni
Researchers at the Center for Language and Speech Processing (CLSP) found that large language models often continue to rely on information learned during training, even when instructed to use new information. The findings could have implications for AI-powered search and question-answering systems, such as chatbots designed to answer questions using newly provided documents rather than prior training.
Findings from the research, presented at the 2026 Annual Meeting of the Association for Computational Linguistics (ACL), explore a fundamental challenge facing modern AI systems: what happens when new information conflicts with the data used to train a language model?
“Large language models are increasingly expected to combine information learned during training with information supplied at the time of use,” says author Kaiser Sun, a PhD student at CLSP. “A model may be asked to answer questions based on retrieved documents, summarize reports, or evaluate content generated by other AI systems. In many of these situations, the information provided to the model may differ from what it learned during training.”
The researchers refer to these situations as “context-memory conflicts,” cases in which external information contradicts a model’s internal knowledge. While such conflicts are common in real-world applications, such as chatbots instructed to answer questions based on a specific document or knowledge base, relatively little research has examined how models resolve them.
To better understand the problem, the team developed a diagnostic framework that created contradictions between information learned during training and new information presented in a prompt. They then evaluated several open-source language models on tasks requiring different levels of reasoning and knowledge use.
“The results revealed a consistent pattern,” says author Mark Dredze, the John C. Malone Professor of Computer Science and the director of the Data Science and AI Institute at Johns Hopkins. “Models performed best when the information in a prompt aligned with their existing knowledge. When the information conflicted with what the models already believed, performance declined.” The decline was even more pronounced when the contradictory information appeared implausible or inconsistent with common expectations.
One of the study’s most notable findings was that language models could not fully suppress their internal knowledge, even when prompted to avoid using such information in developing responses. In tasks where models were told to answer questions using only information contained in a provided passage, they still showed a tendency to rely on previously learned information.
The researchers also tested whether giving models an explanation for why newly provided information conflicted with what they had previously learned affected their responses. When conflicting claims were accompanied by a rationale explaining the discrepancy, models were more likely to rely on the new information.
The findings extend beyond question answering, say the researchers. “Language models are increasingly being used to evaluate the outputs of other AI systems, acting as automated judges in research benchmarks and applications,” says Dredze. “Our study found that a model’s own knowledge can influence these evaluations, potentially causing it to reject answers that are correct within the provided context simply because they conflict with the model’s internal beliefs.”
“Future AI systems may need to do more than retrieve information,” says author Fan Bai, a former postdoctoral fellow in Dredze’s lab at Johns Hopkins. “They may also need better mechanisms for reasoning about conflicts between prior knowledge and newly presented evidence.”
The research was supported in part by the Defense Advanced Research Projects Agency (DARPA) through its SciFY program.