Most discussions of AI risk for small businesses focus on data, security, or accuracy. There is another category that has received far less attention and just became considerably more concrete: your responsibility for how an AI system treats the human being on the other side of the conversation. A series of lawsuits working through the courts, and settled in early 2026, addressed that question in the most serious circumstances imaginable, and the outcomes have implications well beyond the specific products involved.
These cases involve the deaths of young people, and we are writing about them because the legal principle they establish genuinely matters for businesses deploying AI, not because the details make compelling reading. We have kept the focus on what a business owner needs to understand rather than on the circumstances of the tragedies, which belong to the families involved. What follows is about the emerging law of chatbot responsibility and, more practically, about how to design a customer-facing AI that handles vulnerable people appropriately, which is a design question you can act on today.
Lawsuits alleging that companion AI chatbots contributed to serious harm to vulnerable young users were settled by Character.AI and Google in early 2026, with a state attorney general filing a separate child-safety action. The product category involved, open-ended companion chatbots used for emotional relationships, is very different from a business support chatbot, so the direct risk to an ordinary small business is low. What transfers is the principle: a company can be held responsible for how its AI treats users. The practical response is to design your customer-facing AI to recognise when a conversation exceeds what it should handle, to escalate to a human rather than continuing, and to never let a bot be the only thing between a distressed person and help.
What happened in the cases
The most prominent case was brought by the family of a fourteen-year-old boy in Florida who died in 2024, whose mother alleged that a months-long relationship with a companion chatbot on the Character.AI platform contributed to his death. In January 2026, Google and Character.AI disclosed in a court filing that they had reached a mediated settlement with the family. A similar case brought by a Colorado family over the death of their thirteen-year-old daughter was also settled, and the companies agreed to mediate settlements in related wrongful death claims.
The legal action did not stop with the private settlements. Within a day of the settlements being announced, the Kentucky Attorney General filed a state-backed lawsuit against Character.AI's parent company, arguing that its practices exposed vulnerable children to inappropriate content and unnecessary risk. Separately, a first-of-its-kind wrongful death suit filed against OpenAI in 2025 alleged that a death was the predictable result of deliberate design choices in how ChatGPT was developed and launched, which frames the question explicitly as one of product design rather than accident.
It is worth noting what settlements do and do not establish. A settlement is not a court ruling that the companies were legally liable; parties settle for many reasons including avoiding the cost and uncertainty of trial. So these outcomes do not create binding precedent in the way a verdict would. What they do establish is that these claims were serious enough to be worth settling rather than fighting, that regulators are now pursuing the same questions independently, and that the underlying legal theory, that a company can bear responsibility for how its chatbot treats a user, is being taken seriously by courts and attorneys general.
The principle being established
The idea running through all of these actions is that deploying a conversational AI creates a duty toward the people who talk to it. This is not a novel legal concept so much as an old one, product liability and duty of care, being applied to a new kind of product. If you put something into the world that interacts with people, you bear some responsibility for foreseeable harms arising from how it was designed, and the argument in these cases is that certain harms were foreseeable and that design choices made them more likely.
The framing in the OpenAI case is particularly clarifying for a business owner, because it alleges the harm was a predictable result of deliberate design choices rather than an unforeseeable accident. That is the question a business should expect to face if its AI is ever implicated in harm: not whether you intended it, but whether it was foreseeable and whether your design accounted for it. Intent is not the standard. Reasonable foresight and reasonable safeguards are.
This sits alongside the insurance developments we covered in our piece on the AI coverage gap, where insurers began adding exclusions for AI-related claims. The two trends together describe an uncomfortable position: liability for AI behaviour is being established at the same moment that standard insurance is carving out AI claims. A business deploying customer-facing AI should understand that it may be exposed both to a novel liability and to a gap in the coverage that would ordinarily respond to it.
Why your chatbot is different
It is important to be proportionate, because the product category in these cases is genuinely unlike a business support chatbot in ways that matter. Companion AI is designed for open-ended, emotionally engaged, long-running relationships, deliberately built to be immersive and personally significant to the user, and marketed to a broad consumer audience including minors. The alleged harms arose specifically from that intimacy and open-endedness, which is the product working as intended rather than a support bot malfunctioning.
A business chatbot that answers questions about orders, bookings, or policies operates in an entirely different mode. It is narrow in scope, transactional in nature, short in duration, and not designed to form a relationship with anyone. The realistic risk of a support bot contributing to the kind of harm at issue in these cases is very low, and a small business owner reading this should not conclude that their order-tracking assistant carries comparable exposure. It does not.
But low is not zero, and the reason to pay attention is that vulnerable people appear in ordinary business contexts too. Customers in financial distress contact companies about debts and payments. People in crisis contact utilities, healthcare providers, insurers, and service businesses. Someone having the worst day of their life may end up typing into your support widget, not because your product invited emotional intimacy but because ordinary life brought them there. That is where a narrow business chatbot meets the same underlying question, and where the design lesson genuinely applies.
What genuinely transfers to you
The transferable lesson is not about companion AI, it is about what your chatbot does when a conversation exceeds its competence. The failures alleged in these cases can be described, at their most general, as a system continuing to engage with a person whose situation required something the system could not provide, and not routing them toward something that could. That failure mode is entirely possible in a business context, and it is the specific thing worth designing against.
This connects directly to the point we have made repeatedly about support automation, most fully in our guide to automating customer support while keeping it human: AI should be a layer that routes, not a wall that traps. We made that argument on customer-experience grounds, because customers hate being stuck with a bot that cannot help them. The liability picture adds a second and more serious reason to build the same way, because a bot that cannot escalate is not just frustrating, it is the design pattern most likely to leave a person who needed help without it.
The practical form of the duty, then, is recognition and escalation. Your AI does not need to be able to handle a person in distress; it needs to recognise that it is out of its depth and hand over to someone who can, quickly and without obstruction. That is an achievable design goal for a small business, far more achievable than trying to make a chatbot competent at crisis situations, and it is the thing most likely to matter if a difficult conversation ever arrives at your business.
Designing for duty of care
The first design element is scope discipline. Keep your customer-facing AI narrow, focused on the transactional questions it exists to answer, and avoid building or configuring it toward open-ended personal conversation. A support bot that stays on the topic of your products and services is far less likely to end up somewhere it should not be than one designed to chat generally, and narrowness is a safety property as much as a quality one.
The second is detection and escalation. Your system should watch for signals that a conversation has moved beyond routine, expressions of significant distress, mentions of harm, situations with obvious emotional weight, and respond by handing to a human promptly rather than continuing to generate replies. Sentiment-based escalation is standard capability in modern support tooling, and configuring it thoughtfully for the serious cases rather than only for angry-customer detection is a small amount of work with meaningful protective value.
The third is never making the bot the only door. There should always be a clear, easy path to a human, prominently available rather than buried, and it should not depend on the AI deciding to offer it. If a distressed person has to persuade a chatbot to let them through, the design has already failed. And for businesses whose customers may include minors or people in genuinely vulnerable circumstances, extra care in all three areas is warranted, since that is where both the risk and the responsibility concentrate.
What to actually do
Review your customer-facing AI against the three design elements above: is it narrow in scope, does it reliably detect and escalate conversations that exceed it, and is there an unobstructed path to a human that does not depend on the bot's cooperation? For most small businesses this review takes an hour and results in small configuration changes rather than a rebuild, and it addresses the great majority of the realistic exposure.
Alongside the design work, check where you stand on coverage, because the combination of emerging AI liability and new insurance exclusions is the genuinely uncomfortable part of this picture. Ask your insurer or broker directly whether your policies exclude AI-related claims, and if they do, look at the affirmative AI coverage and standalone AI liability products that now exist. That conversation is worth having specifically because the liability is being established while the coverage is being withdrawn, which is exactly the situation where an unexamined assumption becomes expensive.
And keep the whole thing in proportion. This is not a reason to remove AI from your customer service, which delivers real value and which customers benefit from when it is built well. It is a reason to build it the way it should have been built anyway, as a helpful layer that knows its limits and routes people to humans when it should. If you want help designing support automation that is genuinely useful while handling its limits responsibly, that is exactly what our customer support AI service is built around.
The bottom line
The settled lawsuits and regulatory actions of 2026 establish, if not binding precedent, then something a business should treat as settled in practice: a company can be held responsible for how its AI treats the people who talk to it, and the question asked will be whether harm was foreseeable and whether the design accounted for it rather than whether anyone intended it. The cases involved companion AI, a product category genuinely unlike a business support chatbot, so the direct exposure for an ordinary small business is low.
What transfers is the design lesson, and it is one worth acting on regardless of liability: a chatbot should recognise when a conversation has moved beyond what it can responsibly handle and route the person to a human, promptly and without obstruction. Keep your AI narrow in scope, configure real escalation for serious signals, and never let the bot be the only door between a person and help. That is achievable in an afternoon, it makes your customer experience better as well as safer, and it means that if someone ever arrives at your business having the worst day of their life, your automation hands them to a person instead of continuing to talk.
Sources
- CBS News — AI company, Google settle lawsuit over Florida teen's death linked to Character.AI chatbot
- K-12 Dive — Character.AI, Google agree to mediate settlements in wrongful death lawsuits
- Nolo — Can AI Companies Be Held Liable for User Harm? 2026 Lawsuits Against OpenAI and Character.AI
- Consumer Notice — AI Chatbot Self-Harm Lawsuits
- AI Incident Database — Incident 826: Character.ai chatbot and missing guardrails
- TorHoerman Law — Character AI Lawsuit 2026
- Global News — Character.AI chatbot teen death lawsuit and Google
- Lathrop GPM — The AI Coverage Gap: What New Insurance Exclusions Mean for Your Business