Large Language Models (LLMs) like Open AI GPT 4o, Llama-2 and Mistral-7b have transformed the way we interact with AI, enabling them to answer questions, assist with creative writing, and much more. However, a significant challenge persists: these models can be unreliable, occasionally generating incorrect information while sounding extremely confident. This phenomenon, often called a "hallucination," poses major challenges for using LLMs in high-stakes or sensitive scenarios. A new study, led by a team of experts from institutions such as SRI and the Army Cyber Institute, is tackling this issue by focusing on uncertainty quantification (UQ) in LLMs. This research aims to enhance the reliability of these models, making them more trustworthy and effective.

LLMs can generate responses with varying degrees of accuracy, making it difficult for users to know when to trust the AI's outputs. The root of the problem lies in how these models are trained and how they produce free-form text, which can be inconsistent in quality. Unlike traditional AI models that work with predefined classes, LLMs produce responses of different lengths and syntactic structures, complicating the task of determining their accuracy.

The uncertainty in LLM outputs can be quantified through various means, such as entropy (a measure of randomness) and self-consistency checks. The idea is that when a model provides multiple similar answers to the same question, it is more likely that the responses are accurate. This research builds on previous work on uncertainty quantification, aiming to refine these methods for better practical outcomes.

The core innovation presented in the paper is a method called dynamic semantic clustering, inspired by the Chinese Restaurant Process (CRP). This technique groups the generated responses of an LLM into semantic clusters based on their meanings rather than just their word forms. By evaluating the similarity between different responses, the model can estimate how confident it should be in the answer it provides.

In practical terms, this clustering helps to determine whether an LLM is providing a variety of plausible answers (indicating high uncertainty) or consistently similar answers (indicating higher confidence). By calculating the entropy of these clusters, the model can quantify its uncertainty, providing valuable feedback to users about how much trust they should place in the output.

Another important aspect of the research involves integrating conformal prediction, a statistical technique that helps models provide confidence levels for their predictions. Instead of giving a single answer, the enhanced LLM can offer a set of possible responses, with a guarantee that the correct answer is included within a specified probability level.

This approach, which uses the likelihood of the semantic clusters as a scoring mechanism, allows the model to offer multiple options when uncertain—a useful feature for scenarios where reliability is crucial. For example, in medical or legal applications, having a set of answers along with a confidence level can help human experts make better-informed decisions, rather than relying on a single, potentially incorrect response.

The researchers tested their approach on well-known benchmarks, such as the COQA and TriviaQA datasets, using LLMs like Llama-2-13b and Mistral-7b. Their results showed significant improvements over existing methods. Metrics such as AUROC (Area Under Receiver Operating Characteristic) and AURAC (Area Under Rejection-Accuracy Curve) were used to assess performance, and the proposed method consistently outperformed state-of-the-art baselines.

One of the highlights of the study is that the conformal predictor generated smaller prediction sets while maintaining the same probabilistic guarantee of including the correct response, compared to baseline methods. This means the model was more precise in its predictions, reducing the number of options without sacrificing accuracy—a valuable improvement for practical use.

This research marks an important step forward in enhancing the reliability of generative AI. By focusing on dynamic semantic clustering and integrating conformal prediction, the team has developed a system that not only understands when it is uncertain but can also communicate that uncertainty effectively. This capability is crucial for making LLMs more reliable, especially in applications where incorrect information can have serious consequences.

The approach could be used across various industries—from customer service chatbots that inform users when a human agent might be needed, to educational tools that provide multiple possible answers, helping learners understand complex topics from different perspectives. The use of clustering and conformal prediction also opens new avenues for research in making AI more transparent and interpretable, key elements for building public trust in AI systems.

Addressing uncertainty in LLMs is not just a technical challenge but a necessity for making AI more reliable and trustworthy. The dynamic semantic clustering and conformal prediction techniques presented in this study provide a promising path forward. As AI continues to evolve, methods like these will be essential to ensure that LLMs can be used safely and effectively, especially in high-stakes situations.

For those interested in delving deeper, the research code is publicly available, providing an opportunity for further exploration and innovation in the field of generative AI.

Reference Kaur, R., Samplawski, C., Cobb, A. D., Roy, A., Matejek, B., Acharya, M., Elenius, D., Berenbeim, A. M., Pavlik, J. A., Bastian, N. D., & Jha, S. (2024). Addressing Uncertainty in LLMs to Enhance Reliability in Generative AI. NeurIPS 2024 Safe Generative AI Workshop. Retrieved from [https://arxiv.org/pdf/2411.02381]