Is a chatbot high risk under the EU AI Act?

Usually not. A customer support or sales chatbot is, in most cases, a "limited risk" system: it must tell people they are talking to an AI. It becomes high risk only when it is used for one of the purposes listed in Annex III of the AI Act.

Not legal advice. Sources: Regulation (EU) 2024/1689 (the AI Act).

What the AI Act says about chatbots

The AI Act sorts AI systems by what they are used for, not by the technology. A chatbot built on a large language model is not high risk because of the model it uses.

Chatbots fall under Article 50(1): the provider must design the system so that people are told they are interacting with an AI, unless that is obvious from the context. The information must be given at the latest at the first interaction, in a clear and accessible way (Article 50(5)). If the chatbot also generates images, audio or video, that output must be marked as AI-generated in a machine-readable way (Article 50(2)).

When a chatbot is high risk

A chatbot is high risk when it is used for a purpose listed in Annex III and none of the exceptions of Article 6(3) apply. Examples:

  • a recruiting chatbot that screens or ranks candidates (Annex III, point 4);
  • a chatbot that decides whether someone gets a loan, or scores their creditworthiness (point 5(b));
  • a chatbot that decides who is admitted to a school or course, or grades students (point 3);
  • a chatbot that decides whether someone is eligible for public benefits (point 5(a)).

A chatbot that only answers questions about these topics (for example explaining how a loan application works) is not doing the listed task. The test is whether it evaluates or decides about people.

When a chatbot is prohibited

Article 5 bans a few practices outright, whatever the interface. A chatbot that uses manipulative or deceptive techniques likely to cause significant harm, or that exploits people's vulnerabilities such as their age, is prohibited. So is inferring the emotions of employees or students, outside medical or safety reasons.

Who has to do what

If you build the chatbot and offer it under your name, you are its provider: the Article 50(1) notice is your duty. If you embed a chatbot product from another company in your website, that company is the provider and you are a deployer: check that the notice is there and that your own page does not hide it. If you put your own name on the product or change what it is used for, you can become its provider (Article 25).

What to do now

  1. Write down what the chatbot is used for, and who talks to it.
  2. Check that it says it is an AI before or at the start of the conversation.
  3. If it touches hiring, credit, insurance, education or public services, check whether it evaluates people: if so, run the full risk check below.
  4. Keep a record of the models, data and prompts it uses: an AI-BOM does this.

Questions

Is ChatGPT high risk under the EU AI Act?

The model behind ChatGPT is a general-purpose AI model, with its own obligations for its provider (Articles 53 to 55). Whether your use of it is high risk depends on what you use it for, not on the model.

Do I need to say that my chatbot is an AI?

Yes, unless it is obvious to a reasonably well-informed person. Article 50(1) requires the provider to design the system so that people are told, at the latest at the first interaction.

Does my chatbot need a CE marking?

Only if it is a high-risk system. A support or sales chatbot that does not evaluate or decide about people in an Annex III area does not.

More guides

Published ; awaiting review.