program-of-thought-b6c5f74f·1 events·first seen Aliases: Program-of-Thought
Researchers introduce CreditCardQA, a 1,800-question benchmark derived from real credit card agreements to evaluate LLM numerical and financial reasoning. Evaluation across multiple models under Chain-of-Thought and Program-of-Thought prompting shows PoT consistently improves performance and narrows open/closed-source gaps. Error analysis reveals failures stem primarily from misapplied financial rules and misunderstood contractual terms rather than arithmetic errors, with edge cases disproportionately affecting financially vulnerable users.