Finance and Accounting Research Seminar - Hui Chen - "Out of the Black Box: Uncertainty Quantifications for LLMs via Conditional Probabilities"

Autoregressive LLMs generate text by sampling from estimated probability dis tributions over the next token, conditional on preceding context. We leverage these conditional probabilities to construct an entropy-based measure of prediction uncer tainty, which we term inner confidence. Using news classification as a testbed, we show that LLM predictions with higher inner confidence are systematically more accurate. To assess the measure’s economic relevance, we build long-short portfolios based on LLM predictions. Conditioning on inner confidence significantly improves the portfo lio performance: high-confidence predictions achieve Sharpe ratios roughly 20% higher than the unconditional benchmark, while low-confidence predictions yield no excess re turns. By contrast, LLM’s self-declared confidence exhibits strong biases and delivers no comparable gains.