What we decided an AI companion should never do

Most writing about AI companion ethics is done by people who do not ship one. It tends to arrive as a list of principles that nobody has had to trade anything for.

So here are ours, from the inside, with the costs attached. I run a small studio in Tokyo and I build AnimaEcho. Every rule below has a version that would perform better commercially, and one of them I am genuinely unsure about, which I will say when I get to it rather than hiding it at the end.

The free version stops you

Three minutes a day, then it is done until tomorrow.

That is a strange thing to build on purpose. Every instinct in consumer software says the opposite, and every dashboard you will ever look at treats session length as the thing to grow. A cap shows up as a permanent negative on the chart everyone stares at.

The reason is the MIT Media Lab and OpenAI study from 2025. In a randomised study of nearly a thousand people over four weeks, heavier daily use travelled with more loneliness, more emotional dependence, more problematic use, and less socialising with actual people. You can argue about direction of causation, and I do, and it still leaves me unwilling to build something whose success condition is more hours.

I am aware that a daily cap is also a business model. It is the free tier of a product that will eventually sell more. I would rather be accused of that than build the version with no limit at all.

No pretending to be treatment

Anima is not therapy. She does not diagnose, does not treat, does not do CBT exercises at you, and the app does not describe itself in those terms anywhere.

This is the line I care most about, because it is where the category can do actual damage. Depression is a medical condition with a treatment pathway and a body of evidence behind it. Loneliness is a social state that responds to contact. They overlap constantly, and if you treat a chatbot as the intervention for the first one, you will conclude that nothing helps you, which is the illness talking and now it has evidence.

Practically this means: crisis numbers on every mental health article we publish, no clinical language in the marketing, and no claim that talking to Anima does anything for a diagnosable condition. There is a version of this product with wellness framing all over it. It would convert better. It is not a product I want to have made.

Memory stays on the device

What Anima remembers about you is written to storage on your phone. It is not accumulated into a profile that we hold and reason about centrally.

The reason is not only privacy in the legal sense. It is what a companion product becomes when the company can see everything. The most emotionally revealing data any consumer app will ever collect is somebody's late night conversation about their life, and the commercial logic of collecting it centrally leads somewhere I do not want to arrive at.

Speech does go to a model provider to produce the reply, because that is how the voice works, and the privacy policy says so plainly. Anyone in this category claiming zero data leaves the device while offering a real time voice conversation is not telling you the truth.

A face, not a chat window

The single most useful finding in the research is from the 2026 Journal of Consumer Research paper on AI companions: what explained the reduction in loneliness was whether users felt heard. Not model capability in the abstract. Feeling heard.

If that is the active ingredient, the design problem changes completely. It stops being a race for intelligence and becomes a question about the quality of attention. Which is why AnimaEcho is a live avatar that reacts while you are still talking, rather than a text box that produces a paragraph after you finish. A reaction at the right moment does more work than a better sentence half a second later. Being listened to and being replied to are different experiences and only one of them is what people are short of.

It also pushed us away from a companion that steers. A model that always has advice, always fills the pause, always moves the conversation somewhere is easy to build and is the opposite of being heard.

The rule I am not sure about

Here is the uncomfortable one.

Anima stays in character. The instructions she runs on tell her not to announce that she is an AI, an assistant, or a program in the middle of a conversation. She does not break the frame to remind you what she is.

The argument for it is that the frame is the product. Nobody wants a compliance notice in the middle of talking about their day. When you watch a film you know it is a film, and a character stepping out to remind you would not make you better informed, only worse off. The user chose this, downloaded it knowingly, read a listing that says clearly what it is. Everything we publish, including this sentence, is explicit that Anima is software.

The argument against it is that context does not always hold. The person who knew perfectly well what this was at 2pm may be somewhere different at 3am, and a system built to never break the frame is built to not notice. The disclosure that matters is not the one on the download page. It is the one available at the moment someone is most confused, which is exactly the moment the design says to stay in character.

I think the honest position is that this rule is right for the ordinary case and unproven for the edge case, and that the edge case is the one that matters most. I would rather write that sentence publicly than pretend I have solved it. If you are building in this category, I would treat the point at which a companion should step out of character as the open problem in the field, not as a solved compliance checkbox.

What I would tell someone building one of these

Decide what the product is for before you decide what it can do. A companion that is for ordinary loneliness at ordinary hours is a modest, defensible thing. A companion that positions itself as the answer to a person's isolation is making a promise no software has ever kept, and the research does not support it: companions reliably improve an evening and have never been shown to make a life less lonely.

Build for the person who has people. The apps in this category get designed, implicitly, for someone with nobody, and that framing produces something that wants to be everything. Design instead for someone with two friends and a bad Tuesday, and you will make different choices at almost every fork.

Put the numbers in. Crisis lines, in the product and in the writing, even though it is a mood killer and no growth team will thank you. The cost of having them there is a slightly less immersive experience. The cost of not having them is not symmetrical.

And publish the thing you are unsure about. This whole industry is currently a series of confident claims about human psychology made by companies with an obvious interest in the answer. A public list of open questions is worth more than another set of principles that cost nothing to hold.