AI Bots Passed the CFA Level III Exam. An Estate-Planning Test Was a Different Story

Psst, what’d you get for question 3?

Nearly two dozen large language models including Gemini, GPT and Llama took a mock CFA Level III exam, often viewed as one of the hardest tests an advisor can take. Scarily, a handful of them actually did very well, according to a study conducted by the New York University Stern School of Business and AI wealth platform Goodfin. In previous studies, the LLMs fared well on the multiple choice questions, but struggled on those pesky essay sections (we hear you). This time around, though, a few chatbots excelled at both.

While that all may sound a little disconcerting, researchers underscored the critical need for continued human oversight in financial decision-making. “We emphasize that current LLMs should be viewed as assistive tools rather than autonomous decisionmakers for financial applications,” they said in the report.

Cram Session

The level III CFA exam often requires hundreds of hours of prep time over the course of months. The exam in February had a 49% pass rate, meaning half the people who took it would have to take it again to get certified. The researchers estimated a passing grade for the exam is 63%. In some cases, AI almost made it look too easy:

• Open AI’s o4-mini model received a composite score of 79%. 
• Google’s Gemini 2.5 Flash model scored a grade of 77%.
• Anthropic’s Claude Opus 4 received a 75% grade. 

While AIs’ strategic thinking is improving, the tools performed best when the questions were simpler and more straightforward. The tests were graded by both AI and humans, and when it came to the essay questions, humans were actually more lenient, awarding an average of 5.6 points more than the LLM-graders. 

Test Time. If you’re still a little worried about AI replacing advisor jobs, this might pick up your spirits. EncorEstate Plans recently tested 46 LLMs on estate-planning questions, and the results were not great. Some platforms failed more than half the time, while others could answer only simple questions. “Clients often turn to AI for quick estate planning advice, but our study shows technology can’t replace the nuanced expertise of human professionals,” Matt Morris, CEO of EncorEstate Plans, said in a statement.

So rest easy knowing AI just can’t match the level of bespoke financial planning and critical thinking offered by humans … yet.