Economics professor finds LLMs promising but premature
Matthew Kovach makes the distinction clear: He is not an artificial intelligence (AI) researcher but a behavioral economist interested in how AI makes decisions. Last year, Kovach and his team sought to find if large language models (LLMs) could replace financial advisers. The conclusion of their research? Not yet.
“This project started when I was chatting with some economist friends of mine,” explains Kovach, assistant economics professor in the Mitch Daniels School of Business. “Language learning models like ChatGPT interact in such a human way, so we wanted to know if they could give personalized investment advice.”
Kovach says people have already been asking LLMs for advice on homework, writing and code, so it was likely they would ask LLMs for financial suggestions, too.
“People are often unsure about how to make investment decisions,” Kovach says. “There is a huge market paying experts for financial advice. So if people are asking LLMs for help, it is important to assess what that advice looks like.”
For the paper “Learning to be Homo Economicus: Can an LLM Learn Preferences from Choice” Kovach and his team posed investment-related questions to ChatGPT, providing the model with known probabilities. These questions came from research papers previously presented to human participants. The team then compared the responses ChatGPT gave with the responses humans gave. The results surprised Kovach — the LLM chatbot seemed to understand a number of complex economic theories some humans struggled to grasp.
“On one hand, ChatGPT is very good at making consistent choices, which I did not expect. Much better than humans,” Kovach explains.
Kovach notes that ChatGPT doesn’t make some of the poor decisions found in the human data. This means LLMs may be useful supplemental tools for decision-making. But he says the technology doesn’t hit the mark at providing personalized advice and tends to make riskier decisions.
“You may think ChatGPT makes safer decisions, but it turns out that LLMs prioritize maximizing the expected value,” he says. “It doesn’t take into consideration the decision-maker’s attitudes about risk.”
For example, ChatGPT tends to suggest investing in the stock market instead of bonds. However, markets fluctuate and are often riskier investments. That’s why many financial advisers suggest diversifying investment portfolios.
“Would I use ChatGPT for financial planning?” Kovach ponders. “Not quite yet. I think it has a lot of potential. I may use it as a way to gather one perspective if I’m deciding between investments, but I wouldn’t rely on it for the final decision.”
Kovach and his co-authors published their paper “Learning to be Homo Economicus: Can an LLM Learn Preferences from Choice?” on arXiv, Cornell’s open-access archive, in January. The team plans to continue designing experiments to test LLMs’ economic capabilities as the technology evolves and learns.
Members of the Purdue community can sign up to receive a monthly email newsletter covering teaching and learning AI topics at Purdue.
Last updated: March 28, 2024
Author: Malini Nair, AI Communication Assistant for Student Success Programs, nair112@purdue.edu