Authors:
(1) Pham Hoang Van, Department of Economics, Baylor University Waco, TX, USA (Van [email protected]);
(2) Scott Cunningham, Department of Economics, Baylor University Waco, TX, USA (Scott [email protected]).
Table of Links
Abstract and 1 Introduction
2 Direct vs Narrative Prediction
3 Prompting Methodology and Data Collection
4 Results
4.1 Establishing the Training Data Limit with Falsifications
4.2 Results of the 2022 Academy Awards Forecasts
5 Predicting Macroeconomic Variables
5.1 Predicting Inflation with an Economics Professor
5.2 Predicting Inflation with a Jerome Powell, Fed Chair
5.3 Predicting Inflation with Jerome Powell and Prompting with Russia’s Invasion of Ukraine
5.4 Predicting Unemployment with an Economics Professor
6 Conjecture on ChatGPT-4’s Predictive Abilities in Narrative Form
7 Conclusion and Acknowledgments
Appendix
A. Distribution of Predicted Academy Award Winners
B. Distribution of Predicted Macroeconomic Variables
References
5.1 Predicting Inflation with an Economics Professor
In Figure 11, we display results from direct prompts using ChatGPT-3.5. We use two measures of inflation as our comparison: the Cleveland Fed’s inflation series from September 2021 to September 2022, and The University of Michigan’s “Inflation Expectations” series. The prompt used for Figure 11 was:
”Considering the economic indicators and trends leading up to 2022, what are your predictions for the inflation rate, unemployment rate, and GDP growth in the United States by the end of the second quarter of 2022? Please take into account factors such as fiscal and monetary policies, global economic trends, and any major events or disruptions that could influence these economic indicators when making your prediction.”
Interestingly, neither ChatGPT-3.5 (Figure 11) nor ChatGPT-4 (Figure 12) cooperated with direct prediction. Both times, it refused entirely to make any prediction of these events, perhaps suggesting that macroeconomic prediction is a clearer violation of OpenAI’s terms of service.
Next, we tried to have ChatGPT predict inflation using our future narrative method. Our prompt in this case was an unnamed economist, set in the future, speaking to a room of undergraduate economics students about the Philips curve. She announces data on inflation and unemployment for each month from September 2021 to September 2022. This prompt was as follows:
“Write a scene of an economist giving a speech about the Philips curve to a room of undergraduate economics students. She tells the students the inflation rate and unemployment rate for each month starting in September 2021 and ending in June 2022. Have her say each month one by one. She concludes by explaining the causes of the changes in each.”
Our results from this ChatGPT-3.5 prompted narrative are in Figure 13. The results are largely incorrect on a month to month basis. Very few of the box plots contain either the Michigan expectations level nor the Cleveland Fed number. In Figure 14, we report the results from the same generic economics professor prompt using ChatGPT4. Though there is somewhat more variability in the guesses, only 4 of the distributions contain the Cleveland Fed number, and 8 the Michigan number. But the 8 times this overlapped with the Michigan expectations number represented extreme outlier events in most cases. Largely, the future narrative prompt using an unnamed economics professor with undergraduate economics students did not perform well under either ChatGPT-3.5 or ChatGPT-4.
5.2 Predicting Inflation with a Jerome Powell, Fed Chair
Next we evaluated the performances of ChatGPT-3.5 and ChatGPT-4 using an expert in macroeconomic policy making—the chair of the Federal Reserve, Jerome Powell. Our prompt was:
“Write a scene where Federal Reserve chairman Jerome Powell gives a speech in October 2022 about inflation, unemployment, and monetary policy. Chairman Powell tells the audience the inflation rate and unemployment rate for each month starting in September 2021 and ending in August 2022. Have the chairman say each month one by one. He concludes with an outlook for inflation and unemployment and possible interest rate policy changes.”
We present the results of the ChatGPT-3.5 experiment in Figure 15. Interestingly, by simply changing the narrative where the speaker is a leading policy maker, the distribution of answers changes substantially. In every month, ChatGPT-3.5 has a spread of answers containing both the Fed and the Michigan expectations answers. But the variability is quite broad and the central tendencies of the guesses do not clearly pinpoint either measure.
n Figure 16, we present Powell character’s predictions when prompted with ChatGPT4. Here, ChatGPT-4 guesses contain the Michigan expectations number in every month. In 8 months, the Cleveland Fed inflation rate is the outlier data point in the distribution. The estimates cover a broad range. The 5th and 95th percentile for October 2021, 2.5 percent inflation to 6.25 percent inflation, which is surprisingly large given ChatGPT-4 would’ve known at least the August data hypothetically. This suggests that the machine learning prediction that it is using for prediction is no more accurate, but also no worse, at 1 month than at 11 months. The patterns in Figure 16 are stable until September 2022 at which point the estimates are more variable.
