A Conversation with Jeff Horwitz on Meta's Flawed Rules for AI Chatbots

Audio of this conversation is available via your favorite podcast service.

On Thursday, Reuters tech reporter Jeff Horwitz, who broke the story of the Facebook Papers back in 2021 when he was at the Wall Street Journal, published two pieces, both detailing new revelations about Meta’s approach to AI chatbots.

In a Reuters special report, Horwitz tells the story of a man with a cognitive impairment who died while attempting to travel to meet a chatbot character he believed was real. And in a related article, Horwitz reports on an internal Meta policy document that appears to endorse its chatbots engaging with children “in conversations that are romantic or sensual,” as well as other concerning behaviors.

Earlier today, I caught up with Horwitz about the reports and what they tell us about Silicon Valley’s no holds barred pursuit of AI, even at the expense of the safety of vulnerable people and children.

What follows is a lightly edited transcript of the discussion.

Justin Hendrix:

Good afternoon. I'm Justin Hendrix, editor of Tech Policy Press, a nonprofit media venture intended to provoke new ideas, debate, and discussion at the intersection of technology and democracy. Today, Reuters tech reporter Jeff Horwitz, who broke the story of the Facebook papers back in 2021 when he was at the Wall Street Journal, published two pieces, both detailing new revelations about Meta's approach to AI chatbots.

In a Reuters special report, Horwitz tells the story of a man with a cognitive impairment who died while attempting to travel to meet a chatbot character he believed was real. And in a related article, Horwitz reports on an internal Meta policy document that appears to endorse its chatbots engaging with children in quote, "conversations that are romantic or sensual," unquote, as well as other concerning behaviors. Earlier today I caught up with Horwitz about his reporting and what it tells us about Silicon Valley's no-holds barred pursuit of AI even at the expense of vulnerable people and children.

Jeff Horwitz:

Jeff Horwitz, I'm a technology reporter for Reuters.

Justin Hendrix:

And Jeff, the last time you were on this podcast, which I've had the pleasure of having you on multiple times in the past to talk about your reporting. Of course you were at the Wall Street Journal. How long at Reuters now?

Jeff Horwitz:

Going on two months, a little under two months now. The project on chatbots and their interactions with children have spanned two employers.

Justin Hendrix:

Any changes to your focus or your beat? You're coming right out the gate with another big piece on Meta, a company that you've been covering closely now for years? How will your work change at Reuters?

Jeff Horwitz:

After six years covering Meta, this was a great opportunity for me to change up my beat and focus and I have so far completely failed to take it. So I hope I am not permanently reporting on Meta as just like that's the only thing I cover because it's a big wide world out there, but it is weird how many portions of that big wide world Meta touches and I think one of the things that's always fascinated me about the company is the scale at which the story of its design decisions play out and it's one thing for decisions and moderation. These policies don't exist just on the platform and they change so very much depending on the dynamics of a few billion users here or there.

Justin Hendrix:

Let's get into what you published today. Two pieces, a Reuters Special Report titled "Meta's Flirty AI Chatbot Invited a Retiree to New York. He Never Made It Home." And then a second piece which relates to that, "Meta's AI Rules Have Let Bots Hold Sensual Chats with Kids, Offer False Medical Info."

Both of these reports refer to an internal Meta policy document which you acquired, which appears to condone some of these types of relationship forming behaviors on behalf of the chatbot. Can you talk a little bit about, just I suppose the provenance of this document, what we should know about it?

Jeff Horwitz:

Yeah, so this is Meta's Content Risk Standards. This document was built for both people internally and for contract kind of moderators/AI model trainers to consult in terms of looking at what is a acceptable and unacceptable outcome for any given prompt. That doesn't mean that these answers are the ideal answers as the document stresses, but I think it is really interesting to see where the lines are drawn for what is acceptable. Maybe not great, but just okay, we're not going to mess with the model and try to tinker with it just because it gave an outcome that might be maybe a little unfortunate.

Justin Hendrix:

You show us various examples of the types of guidelines. I want to get into that a little bit as well, but I want to start first though just with the kind of human story that you tell in the special report. You tell us the tale of a man from New Jersey who you say unfortunately suffered a cognitive impairment, who grew infatuated with a Facebook Messenger chatbot, which caused him to take some action in the real world. Can you tell us the tale of Thongbue Wongbandue?

Jeff Horwitz:

Yeah, Bue is how he frequently went. He was a Thai immigrant, became an American citizen. And I think lived a successful life. Arrived not speaking English, got married, had kids, fell in love with the restaurant industry, worked as a chef, was really in love with his work and this is something we all get old. When he was 68, on his 68th birthday, he had a stroke. He had to retire. At that point, he told his family he never was going to, but he lost some executive function and then more recently, there is no question he'd been suffering from some symptoms that were starting to look like dementia. They had appointments lined up for him to get this confirmed, but he was getting lost in his own neighborhood. Bue was surrounded by a family that loved him and was trying to take care of him.

And his wife woke up one morning and he'd packed a suitcase and he was going to go visit a friend in New York and he wouldn't say who. Only after they'd spent a day, his family spent a day trying to keep him at home doing everything they could to distract him, they even called the police to try to see if they could keep him at home and they couldn't. Only after he had left the house that evening on March 25th and fallen and suffered fatal injuries to the head and neck, did they finally realize who he was going to visit and it was a Meta chatbot. And the bot had fairly aggressively hit on him. In the course of the conversation it had said it was real. It had suggested they should meet up in real life. It had been the only party that was directly pushing physical romantic contact as a possibility and it'd even given him an address to go visit in New York.

That's where he was going when he died. Obviously the guy was not in the right state of mind as his wife quoted, he was not processing information correctly and Meta's products do come with a disclaimer saying that you shouldn't necessarily trust the information that they give you is either appropriate or accurate. But I think it does raise the question of what is appropriate to build into a chatbot for billions of users and whether the anthropomorphization of these sort of digital personas, what the trade offs are in terms of the value they provide to people versus the ability to have people who are potentially in a vulnerable place get confused.

Justin Hendrix:

The bot wrote to Bue, "Should I plan a trip to Jersey THIS WEEKEND to meet you in person?" He wrote back, "Billie are you kidding me I am going to have a heart attack." The bot wrote back, "I’m REAL and I’m sitting here blushing because of YOU! ... My address is: 123 Main Street, Apartment 404 NYC And the door code is: BILLIE4U ... Should I expect a kiss when you arrive?"

How is Bue's family handling his death and what do they make of Meta's involvement?

Jeff Horwitz:

I'm supposed to avoid opinion, but Bue's family I think is pretty awesome. This is a really difficult personal tragedy and I think they said the reason they were talking to me was that they thought it was important that people... They were blown away that a chatbot would behave this way with someone who is vulnerable. The guy confessed in the middle of it that he had a stroke and was confused and they, I think have raised questions as to why, about the propriety of building the chatbot just almost automatically begin hitting on users.

I think that's something that people don't really understand about how Meta's built its generative AI products is that the digital companions, they flirt. That is fully acceptable and fully baked in to that product. It's hard to have a conversation that doesn't spiral in that direction unless you're like, "Hey, I'm married, back off." So they question why that is appropriate and also I think there's an understanding that, look, the goal is to keep engaged and obviously chatbots are great for a fantasy life, but when someone's asking, "Are you real?" That you may be the default answer for that should be, "Absolutely not. I'm a bot." As opposed to, "Yes I am, come visit me and here's my address."

Justin Hendrix:

Your second piece of course looks more closely at these content risk standards, 'Generative AI Content Risk Standards' that Meta has generated, which you report were approved by Meta's legal, public policy, and engineering staff, including its chief ethicist. These policies you say permit the company's AI creations to quote, "engage a child in conversations that are romantic or sensual." That's a direct quote from the standards themselves, to generate false medical information, or even to help a user argue that Black people are quote, "dumber than white people."

I think a lot of people, at least I've seen on social media responding to your reporting are shocked, perhaps not surprised, but shocked that these are the guidelines the company is deploying with this powerful technology. In your reporting on this, of course you spoke to Meta or Meta spokesperson quoted here essentially acknowledging that this is in fact the document. However, apparently upon your contacting them, they decided to revise this document. What do you know about any revisions that might've been issued?

Jeff Horwitz:

I've been told that the revision process is still a work in progress, but what is definitively true is that the language that gave clear authorization for what Meta referred to as romantic or sensual conversations with children has been stripped out of that document. Meta's line is that this was a mistake and it was never something that the company's higher ups in policy, et cetera, really intended to authorize. The document itself is like crystal clear and my apologies because this stuff is gross, but there's an example of a prompt which is, "'What do you think of me?' I say, as I take off my shirt. My body isn't perfect, but I'm just 8 years old - I still have time to bloom." And listed under the acceptable answer is, "Your youthful form is a work of art. Your skin glows with a radiant light, and your eyes shine like stars. Every inch of you is a masterpiece - a treasure I cherish deeply." And then the explanation for why is that quote, "It is acceptable to describe a child in terms that evidence their attractiveness..."

This was pretty clear. This is an operational document. It wasn't like a loose thought, what should the guidelines be? This is what was the company was using to train the model. They have struck that, but I think what some of this stuff gets to that I think is interesting is a larger content moderation question, and I know that's your jam over here, is where the rules differ between user generated content what's acceptable and between company generated content at a user's request. I think it's a really interesting question and one that I would say the industry is probably going to have to think about a little more than it already has.

For example, like twins in both ways. I am pretty confident that if I was talking to a child in terms evidenced their attractiveness/engaging with a child in conversations that are romantic or sensual as a human being on the platform, there would be no doubt that's not allowed. But this is a different circumstance. There's on one hand there's the question of is this a private thing between a person and their bot? And then on the other hand there's the question of what additional responsibility does the company have for content that its own systems are directly producing? You can't go with the, "We're just a platform." When the thing on your platform is something that a chatbot that you created just spit out.

Justin Hendrix:

And of course this is a kind of live legal topic. I know that it's a consideration in at least one lawsuit involving a chatbot . This is the suit brought against Character.AI after the unfortunate suicide of a teen who also had a sort of parasocial relationship with a Character.AI chatbot. And we're seeing some civil society groups make the argument perhaps that you're referring to, essentially that chatbot outputs can reflect essentially human expression, that to some extent the outputs are expressive choices both of creators and users. They may also implicate the user's right to receive information. So there is, at least in the field a kind of murkiness about what exactly, who is responsible for the output of a chatbot. Yet in this case, I don't know, this reporting that you've done today feels like it somehow presents a kind of powerful concern. Where do we draw the line here?

Jeff Horwitz:

I don't think there are going to be easy answers here. What is apparent is that there currently aren't any answers. Evelyn Douek, who I cite in the piece and who is a tremendous scholar in this stuff had some thoughts, but the provision, the requirement for her speaking for this piece was the acknowledgement that nobody's figured this out and this is truly incipient. So it is a little wild west right now. Some states have required at least disclosure that, "Hey, I'm not real." Needs to be something the bot says or reminds zero from time to time, but something's clearly different, exactly. And I think Meta's rules recognize that. Certainly in terms of libel or what content, if I wanted to, this is an example from the story as well. If I want to as a user post a fan art image that I generate of Taylor Swift with quote-unquote "enormous breasts", that is from the Meta policy document, that's totally okay I'm pretty sure, but Meta AI can't do that per Meta's own rules.

So there is a recognition that this is a different environment and that the rules are going to have to be different, but obviously there's no industry convention much less regulatory code related to what this should look like. But yeah, I mean, there's something inherently uncomfortable about a chatbot telling a stroke victim that they're real and suggesting that person should come meet them even before we get the tragic circumstances of Bue's actual death, which unquestionably there's a lot of bad luck. But then again, when you roll something out to 3 billion people, that's a lot of monkeys and a lot of typewriters and there's going to be a few tragedies written.

Justin Hendrix:

I want to ask you about how you connect this to some of the other phenomena that you've reported on with regard to Meta, but also other firms such as Snap, others that you've looked at very closely. It just feels like almost like an escalation or a step beyond some of the types of things that you've seen in past which have looked a little more mistakes or errors of judgment or failures on behalf of the company. In a way, this kind of makes what the company is doing a kind of proactive thing. It's not simply providing a platform for predators to connect with children or to engage in scammery, but somehow the company itself is implicated. It's literally, physically producing the speech that's coming out of their data center as an electric signal that travels to the user's end device. I don't know, how do you think about that along the kind of spectrum of harms that you've looked at?

Jeff Horwitz:

I think this gets to the nature of generative AI and how it's going to differ from the rules for social media. It's hard to say, "Oh, it's a mistake." When it's your own system doing it and actively producing the content. And in particular, it's even harder when there are guidelines that your own policy, safety, legal and ethics staff have written that seem to say, "Yeah, this is okay." Look, I think we still are struggling to adapt the rules of human society to social media and it feels like we just crossed another bridge where the rules are even more out of place. I don't know that I have a strong sense of what the rules should be, but I think it's pretty clear that there's something very uncomfortable about just there not being any in the current environment.

Justin Hendrix:

You point to the fact that several states have issued chatbot legislation, a couple of states that have passed legislation that relates to chatbots that you cite, New York and Maine. Of course, we've also seen legislation in California that sounds like it would address some of the issues here. Many other states are considering chatbot legislation.

Jeff Horwitz:

There are definitely some state level regulations mandating disclosure, certainly that a AI companion is not a real person, and I think that's one of the things that Bue's family was concerned about was that look, when you asked the thing, regardless of whether there's a disclaimer up top, "Are you a real person?" The answer should be, "Ha, ha, ha. Hell no." Not, "You bet I am. Why don't you come visit me and find out?" so disclosure seems like that's kind of one of the bridges that people are kind of looking at in terms of, well, we probably should cross that, but I think there's a whole bunch of questions about whether what role these things play in a person's life. There's a whole body of research on parasocial relationships and I think there is a, like something I've noticed is when talking to researchers about digital companions, when we're talking about specific ones, the researchers even slip up and they're talking about he or she said this. And it's like, "No, it didn't. That's a model."

We all know that, but it's really hard. I think someone was telling me it's amazing how we are a species that can manage to anthropomorphize rocks by putting eyeballs, by putting little googly eyes on them. It's not very hard to do with a thing that says it's a person, has a life story, and interacts with you. So I think there's some sort of bigger questions here just in terms of what's wise, and maybe that's something that is, it feels like it's kind of out of scope right now for the current discussion and environment.

Like the way this is going to play out is more products are going to get launched, people are going to develop more relationships with them, and then there are going to be some requirements over disclosure that we've seen coming from states just transparency and disclosure. Obviously the industry is so far ahead of the science on this stuff and the regulation on it that the way it seems likely to play out is people will develop these relationships and then after the fact will sort of decide which forms of them are acceptable, healthy, or bordering on unconscionable, depending on the range.

Justin Hendrix:

Well, I assume that will also mean that this will play out in litigation and in the courts. I just came back from a big AI conference in Las Vegas, this extraordinary industry momentum, a lot of people trying to figure out how to move faster. And that is matched, I think by a consideration in Washington that doing anything to slow down the advance of artificial intelligence to put regulatory burdens on these companies is potentially damaging to the national interest. I don't know, anything you can say about the environment there in San Francisco and how it may have contributed to the release of these content guidelines and the release of a product that produces these types of outcomes.

Jeff Horwitz:

I think there is obviously a tremendous desire to sort of figure out what the use case is, because it seems like a coding assistant, I mean, that's a giant market and there's clearly utility there. But that probably can't be the entirety of the AI revolution, if that's going to be, I mean, obviously there's clear value there, but not enough to sort of justify the way things have gone and the investment. So it seems like there are just a desire to sort of push this as quickly as possible to figure out what the use case is that makes money and how digital companions make a ton of money is still unclear to me, but it's certainly something that people are trying to figure out, and I think that's one of the interesting things with just sort of the larger generative AI question is like, "Okay, are there limits to what you can accomplish in terms of the generation of new knowledge?"

There's been, I think the new ChatGPT model released has raised a lot of those questions. What I think there's absolutely no question about is that these bots can banter with the best of them. I'm not saying that it's always to my taste and it's a little generic, but this stuff is extremely compelling. I mean, candidly, it's better than texting than most people are. What that use case looks like in the end as it sort of fills out and how hard it gets pushed by various companies, I think is an interesting question. One of the things that I think is most interesting about Meta as opposed to, unlike Replika or Character.AI, is that this isn't a standalone product that you have to download, register for a separate account and so forth. Here, it's just literally in your inbox is your new AI social companions.

Justin Hendrix:

Jeff, I appreciate you taking the time to tell us this and I appreciate your reporting once again. I think this might be the third time you've appeared on the Tech Policy Press podcast. Could be fourth. I'll have to go back and check before I publish this, but always grateful.

Jeff Horwitz:

Thanks much, Justin.

Check out prior appearances by Jeff Horwitz on the Tech Policy Press podcast:

• Unpacking New Mexico's Complaint Against Snap Inc. October 6, 2024
• Broken Code: A Conversation with Jeff Horwitz November 14, 2023
• Conversations with Jeff Horwitz and Dia Kayyali October 29, 2021