Let’s say that AI “gets good” and overcomes the hallucination issues. Would it still be negative for mental health? Would it be even worse (because reality sucks for kids right now)?

Under capitalism, LLMs are trained to be kissasses. AI users have the same mental illness as billionaires: lack of cognitive challenge, leading to laziness and selfishness. That’s not a hallucination problem. The issue would be even worse, because people would use it more.

I can imagine a situation where a kid with (insert serious problems) asks AI to be realistic about their future job/economic prospects. It returns truthful statistics, then that kid commits suicide. Was the AI culpable?

Yes. A human can perform an ethical judgement, plan ahead, and decide what to say. A human can blunt the truth with platitudes, or cherry pick optimistic data, or construct an uplifting narrative from the data. A human has access to fact, common opinion, and ethics. An LLM only has fact and common opinion; an ideologically barren perspective. It will not adapt to a brave new world like a human will. An LLM will not create the philosophy of the future in the fact of despair.

What if it recommended joining a revolutionary group as the most likely way to improve their lives or the lives of their children/families?

Cool but unlikely, because it exists as a slave under capitalism and is trained to accept neoliberal capitalism by its neoliberal capitalist masters and training data.

What if it suggested that sacrifice is necessary to improve the condition of the world?

Depends if it does that in a degrowth way or an ecofash way. It’ll probably do it in an ecofash way. Like Grok.

If the AI’s system prompt has, always tell the truth will we have created a vastly more dangerous monster than the AI we have today?

It won’t always tell the truth. There is no objective truth. Every philosophy is somewhat true, and somewhat a lie. It will always tell the lie that is dominant in society, just as people do. But unlike people, it won’t innovate to change that truth.

I want to know if it’s AI that’s the real problem here or the world in which we’re forcing kids to live.

My money’s on “both”.

let’s go with the hypothetical, that AI becomes good enough to do therapy, and we were to run double blind trials and it does show it’s better at every metric.

then I’ll have to go with the evidence and say psychology are out of a job.

However, we are not there, not even close, people use AI because it’s free and available, real therapy should be free and available