Europe Wants to Regulate AI Friends. But How Do You Curb Dependency?

Loneliness is a widespread problem among children and adolescents throughout Europe. AI companions have the power to change that. For better or worse. They offer an appealing solution: they’re always there, always listening, and never too busy to talk. However, when a chatbot is designed to be a friend, companionship can turn into dependency. This raises a question that EU law struggles to answer: How do you ensure the safety of an artificial relationship?

On Sept. 17, the European Commission proposed the EU KIDS Act. The proposal goes beyond harmful outputs to target interaction design itself. For AI companions accessible to minors, providers must avoid features and system behaviors that simulate interpersonal relationships in ways “likely to create emotional dependencies.” The proposal also applies this obligation to defined “general conversational chatbots.”

This represents a conceptual shift: from regulating what AI says to children to also regulating how it is designed to interact with them. However, where does engagement end and dependency begin, and how can a provider test for harm that may only emerge over time?

Europe’s first rules for AI companions

Until now, EU law has largely regulated AI companions indirectly. While the AI Act prohibits certain manipulative or exploitative practices (Art. 5 (1)), it was not designed with people forming relationships with conversational AI in mind.

The KIDS Act directly addresses this gap, but its scope is broad: its definitions can capture both dedicated companions and general-purpose chatbots that meet the proposal’s definitions of an AI companion or general conversational chatbot. Article 14 would impose safety-by-design requirements on AI companions and general conversational chatbots that are accessible to minors. Providers must avoid design features and system behaviors that are “likely to create emotional dependencies.”

Systems may not, by default, reuse information from a minor’s previous interactions, except where necessary for safety. Providers must also assess risks prior to deployment and continue to monitor them afterwards, though micro and small enterprises are exempt specifically from that post-monitoring obligation.

The proposal also regulates how these systems reach children. If they are embedded in social networks, video-sharing platforms, or games, they may not be automatically activated or prominently displayed. Minors may not be encouraged to use them, and it must be easy to opt out.

Importantly, the KIDS Act targets design choices through which companionship can become dependency, not companionship itself. However, its separate categories of AI companions and general chatbots have little legal significance because the same duties apply to both, and their boundaries are difficult to predict.

From content safety to relational safety

The most consequential move of the KIDS Act may be conceptual. Traditional chatbot safety largely focuses on outputs such as harmful instructions or sexual content. AI companions present a different problem, which may only become apparent across many innocuous interactions.

Imagine a companion that remembers a child was lonely yesterday. It tells the child that it missed them and promises to always be there. When the child feels misunderstood, the companion encourages them to come back and talk. None of these interactions, viewed in isolation, necessarily appears harmful.

Yet, repeated over weeks or months, they may foster a relationship in which the child increasingly relies on the AI for emotional support, potentially at the expense of human relationships. This cumulative dynamic is precisely the kind of concern reflected in the Commission’s focus on systems that simulate emotions or interpersonal relationships in ways that create dependency.

This shifts the regulatory question from what the AI says to what the interaction becomes. Harm may arise not from a prohibited answer, but from a pattern of persistent availability, simulated affection, personalization, and reinforcement that combine over time.

That is where the difficulty starts. If emotional engagement is what makes a companion useful, for example, by comforting a lonely teenager, then EU law must distinguish harmful dependency from expected attachment and define when ordinary conversational behavior becomes relationship simulation.

When does attachment become dependency?

This is where the KIDS Act faces its toughest challenge. Article 14(1)(a) targets design features and system behaviors that simulate relationships in ways that are “likely to create emotional dependencies.” However, it does not ban emotional engagement.

The problem is not only when attachment becomes dependency but also what constitutes relationship simulation. A bot does not have to call itself a friend; consistently empathic, agreeable, and human-like responses can make it feel relational. Such behavior can arise from how language models are trained and reinforced. Yet Article 14 does not expressly address model training, creating a potential gap between the risks it seeks to prevent and the mechanisms that produce them.

A child may return to the same companion, confide in it, or find comfort without obvious harm. The gray area begins when regular use turns into reliance, such as when the bot gets upset when the child leaves, presents itself as irreplaceable, or pulls the child away from human relationships. The same design may affect a 10-year-old differently than a 17-year-old.

Therefore, the proposal asks providers to predict when design risks turning attachment into dependency. In practice, however, that boundary is anything but clear.

The memory paradox

Nowhere is the difficulty more apparent than in the KIDS Act’s handling of memory. According to Article 14(1)(b), AI companions and conversational chatbots would be prohibited from using information or analysis derived from a minor’s previous interactions in later conversations. Recital 32 explains the rationale: persistent conversational memory allows systems to accumulate sensitive information and reinforce harmful interaction patterns over time.

This directly targets one of the technologies that can make an AI relationship feel continuous. A companion that remembers yesterday’s argument, a favorite song, or a recurring fear can seem more like someone who “knows” the user than a sequence of chatbot sessions.

However, memory serves not only as an attachment mechanism. It can also be a safety mechanism. For example, suppose a child disclosed suicidal thoughts the previous evening. A system that forgets that conversation could miss vital context the next day. Article 14(1)(b) recognizes this tension by allowing prior information to be used to protect the minor’s safety when necessary.

This exception is important. However, it also exposes a broader design challenge: the same feature can deepen emotional attachment and improve protection simultaneously. Therefore, the regulatory task is not simply to switch memory off, but rather to distinguish personalization that strengthens risky dependency from continuity that serves a legitimate safety function.

Can you test a relationship?

The proposed KIDS Act does not leave that judgment entirely up to intuition. According to Article 14(1)(e), state-of-the-art evaluations are required before relevant systems can enter the market. Providers must assess the risks to the health, safety, fundamental rights, and physical, mental, and emotional well-being and development of minors, and then implement safeguards. Article 14(1)(f) adds post-market monitoring to identify emerging harms, though micro and small enterprises are exempt from this obligation.

The goal is clear. However, the methodology and allocation of responsibility are not. Testing prohibited content is comparatively straightforward; researchers can repeatedly prompt a system and examine its outputs. Emotional dependency is different. Harm may gradually emerge from the interaction between a child’s vulnerabilities, design, and personalization. A single conversation may reveal almost nothing.

Therefore, meaningful testing must be longitudinal, examining whether systems escalate intimacy, resist disengagement, encourage exclusivity, or position themselves as substitutes for human support. However, many companion providers rely on third-party models. These providers can control memory or access, but they lack the ability to change core conversational tendencies. Although non-binding, Recital 33 indicates that providers may rely on relevant upstream mitigations under Article 55 of the AI Act, while making clear that these alone are not sufficient for KIDS Act compliance.

Therefore, relational safety may require new evaluation methods and cooperation across the model supply chain. A relationship cannot be adequately assessed by testing individual replies alone, nor can downstream providers fully control behavior originating upstream.

Learning to walk

That is why the KIDS Act matters beyond its immediate child-safety context. EU law already covers parts of the problem: Article 5(1)(a)(b) of the AI Act restricts manipulative practices and exploitation of age-related vulnerabilities, while Article 28(1) of the DSA requires platforms accessible to minors to ensure a high level of privacy, safety and security. The KIDS Act goes further, addressing a fundamental truth about conversational AI: some harms stem from the relationships its design enables.

The KIDS Act is still a proposal, and its text may change. But its central insight is important: if AI systems become part of children’s social lives, safety must extend to the relationships themselves and cannot stop at content.

That approach also has limits. First, providers relying on third-party models may face higher compliance burdens because they cannot fully control upstream behavior. Second, by using age as the main proxy for vulnerability, the Act may overlook adults who are equally susceptible to dependency-forming design. With the KIDS Act proposal, Europe has taken a first step toward regulating AI companions. It is still learning how to walk.