California Bar Admits to Using AI to Develop Bar Exam Questions

In a press release published Monday, the State Bar of California acknowledged that it, through one of its partners, had used generative artificial intelligence (AI) to develop 23 of the 171 scored multiple-choice questions on the bar exam.

The State Bar is the administrative arm of the California Supreme Court. It also administers the state’s bar exam, which prospective lawyers must pass before being allowed to practice law.

However, the February test was controversial even before it started. The state had previously decided not to use the national bar exam developed by the National Conference of Bar Examiners. It had also opted to move to a hybrid in-person and remote testing process for the February exam.

Unfortunately, that test was beset by many problems. Technical issues plagued the exam, with many students unable to log on at all. Still others faced challenges while taking the exam. As such, the press release asked the California Supreme Court to lower the threshold for passing in fairness to students who had technical issues.

However, the AI acknowledgment has gained the lion’s share of attention. Though just a blurb in the middle of the press release, it has drawn nation-wide criticism.

So what exactly happened and why are so many upset? It turns out to be fairly straightforward.

How The California Bar Used AI

The California bar exam has 171 multiple-choice questions. Of those, 100 were developed by Kaplan, a large company specializing in developing various tests. Another 48 questions came from a first-year law student exam, though the press release doesn’t say which one.

It’s the final 23 questions that’s the focus of the controversy.

ACS Ventures developed those questions. The organization serves as the State Bar’s psychometrician, meaning they specialize in creating and administering psychological tests.

The ACS questions were “developed with the assistance of AI and subsequently reviewed by content validation panels and a subject matter expert.”

The dismay is that the State Bar of California allowed non-lawyers to use AI to draft questions for the bar exam.

However, the bar argues that the questions underwent the regular vetting process. This included vetting by humans and panelists at the State Bar itself.

Unfortunately, that hasn’t done much to placate criticism. Many, including applicants and law school professors, are upset and are making their displeasure known.

A Question of Transparency

While there are many issues with this, one of the largest is transparency. Simply put, no one, not even the California Supreme Court, was aware of the usage of AI before this press release.

Only after technical issues forced the State Bar to seek a scoring change did it reveal the use of AI. If the test had gone off without a hitch, we might never have found out.

According to Alex Chan, the chair of the State Bar’s Committee of Bar Examiners, said that the California Supreme Court asked the State Bar to explore new technologies, including AI, to improve testing and lower costs.

If so, why not be transparent with the California Supreme Court and the public? If the use of AI has no impact on the test and can save money, what is lost by disclosing it in advance?

The answer is clear. They know that AI is controversial and that they would face criticism for using it. The State Bar, through its partner, opted to make use of AI without disclosing it.

This has angered a lot of people, and rightfully so. The State Bar is taking a lot of big swings right now. For better or worse, it’s shaking up the state’s examination process. Above all else, that process needs to be transparent and open to input and criticism.

When it comes to using AI, they failed to provide that transparency. No matter where one stands on AI, disclosure and transparency are key to using it ethically. In many ways, the failure to be transparent is as bad, if not worse, than the use of AI itself.

Bottom Line

State bars tend to be very conservative organizations when it comes to technology. Their focus is on ensuring their field’s integrity, which involves a certain resistance to change.

However, the State Bar of California is making major changes. Those changes, according to most reports, are largely about saving money. This includes decisions to cut exam locations, offer remote examinations, creating a new test and the use of AI.

All of these steps are controversial. However, sometimes distressed organizations have to make difficult and unpopular decisions. That said, those decisions should be transparent, especially when those decisions have a significant impact on both the legal field and the public.

While the debate about AI will continue, the California State Bar let everyone down by not properly disclosing the use of AI. Even if they fully vetted the questions, applicants and the public deserve to know how the State Bar developed the test.

As a non-lawyer, I will not tell the State Bar Association how they should draft their test. That is well beyond my area of expertise. But, as a member of the public, I depend on lawyers to be knowledgeable, ethical and professional. I have a stake in the outcome of the bar exam.

As such, the State Bar owes the public transparency. That includes disclosing the use of AI for test questions.

It feels like the bare minimum that the State Bar can do…

Header Image: Tobias Kleinlercher / Wikipedia, CC BY-SA 3.0, via Wikimedia Commons