Will AI Sycophancy Contaminate Law Enforcement?

Jake Laperruque is a fellow at Tech Policy Press. This is the first post in “Perils of AI in Policing,” a three-part series on emerging risks of artificial intelligence’s use in law enforcement.

Society’s growing reliance on artificial intelligence has given rise to concerns about AI sycophancy. This dark pattern—the tendency of AI systems to tailor responses to flatter users and validate their point of view—has alarming social ramifications: commentators worry it could warp human values and expectations or prop up reliance on digital yes men, and it's been documented amplifying dissociations and delusions. Beyond these general risks, we need to account for problems AI sycophancy may cause when the tech is applied to specific, high-risk functions. In particular, we ought to examine how it could impact the use of AI in policing.

AI is becoming incorporated into an array of law enforcement tasks as a virtual assistant. Companies like Axon are pushing AI services that scan body camera footage and write automated police reports. Vendors like Truleo promote AI systems serving as a virtual detective, taking investigative case files and providing summaries, timelines, surfacing leads, and providing analytics. The company prompts police to take a “scan the case - what am I missing?” approach with its systems, promising an AI system that “summarizes, organizes, and provides critical insights or gaps.” These tools are being pushed to prosecutors as well as investigators: Thomson Reuters offers a CoCounsel Legal AI tool for prosecutors to summarize evidence files, review discovery materials to “identify key facts” and “create comprehensive timelines” for cases, as well as conduct legal review on evidentiary questions and even draft charging documents and plea agreements.

Could these tools be susceptible to AI sycophancy? This trend can result in outright hallucinations, slanted and selective information, or responses that are merely unctuous and fawning. Sycophancy can occur for numerous reasons, but especially because people tend to gravitate toward agreeable answers, even if unconsciously. This affects model training and user-driven reinforcement learning: users tend to give higher ratings to sycophantic AI systems and use them more frequently, so systems adapt to become even more agreeable. As one research study aptly puts it, “The very feature that causes harm also drives engagement.” This is especially worrisome because it not only warps AI but also creates a feedback loop, where increased use augments AI sycophancy and AI sycophancy encourages more reliance on the software.

It's frighteningly easy to imagine how this could occur during law enforcement use of AI. Officers could be more prone to use an AI police report that depicts their conduct as calm, smart, and professional. Investigators will be more likely to rely on case summarization tools that affirm their suspicions and frame their work as sound. Prosecutors could gravitate toward AI systems that provide straightforward solutions and present favorable assessments of key factual and legal questions for cases.

The impact in each of these contexts could be immense. What if automated police report tools develop a tendency to tailor descriptions of encounters to make police Terry stops and use of force justified by default? How will case summarization tools shape evidence if they are driven to providing investigators with a culprit? If prosecutorial AI assistants are shaped to please their users, will they be overly restrictive in what exculpatory evidence they label as Brady materials that must be disclosed to defendants? Even outside of more severe cases in which evidence could be fabricated, altered, or omitted, AI sycophancy merely slanting summaries to be more favorable to users—in this case, police and prosecutors—could have serious ramifications. And these distorted results will be presented as coming from an unbiased, objective source.

The most basic requirement to address the danger of AI sycophancy in policing is strong transparency. We need independent evaluation of AI systems used by police to test for sycophancy (as well as for general review of potential bias and accuracy issues). This is a serious problem, because we’re likely to confront secrecy from vendors and police alike. Axon’s “Draft One” automated police report drafting tool is already mired in significant transparency problems: Its systems aren’t independently reviewed or tested, and it's designed to avoid tracking which parts of automated police reports are edited by officers. This means there’s no way to audit systems to review how often and in what way this AI system makes errors. And Draft One’s lack of transparency is hardly an isolated incident: there's a long history of the vendors of AI tools for police—from Clearview AI to ShotSpotter (now SoundThinking) to Amazon Rekognition—pushing back against independent testing that is critical of their products, a problem likely to recur since sellers have a financial interest in marketing their products as cutting-edge and effective. Lack of transparency from police is also a proven problem: From facial recognition to automated license plate readers, we’ve seen police hide their use of AI tools as a deliberate tactic.

Cities, states, and lawmakers in Congress need to take a hard, critical look at how AI sycophancy and other unintended consequences of the technology could imperil policing and harm civil liberties and public safety alike. The best course of action may simply be to hit pause on their acquisition and use, especially for tools that haven’t been tried out in less high-stakes arenas. At a minimum, demanding independent review and rigorous, ongoing vetting for any police AI tools is essential. It’s the only way to ensure police do not end up relying on a digital yes man that puts the public at risk.