Extraordinary claims require extraordinary evidence. As far as I can tell, the whole idea of LLM consciousness being a world-wide threat is just something that the hyper-rationalists have convinced each other of. They obviously think it is very real but to me it smacks of intelligence worship. Life is not a DND game where someone can max out persuasion and suddenly get everyone around them to do whatever they want all the time. If I ask Bing what I should do and it responds "diet and exercise" why should I be any more compelled to follow its advice than I do when a doctor says it?
I don’t think people are afraid it will gain power through persuasion alone. For example, an LLM could write novel exploits to gain access to various hardware systems to duplicate and protect itself.
