> Any AGI that is a threat to humanity has to be suicidal to act on it because of how the world works
A "threat to humanity" need not mean "killing all humans next year". It could just mean some entity gaining permanent control. That situation would be irreversible and humanity would be at the mercy of that entity. Chances are, the entity would have emerged victorious from an evolutionary struggle for power and thus would not care about human flourishing.
That entity need not be a single AI. It could be a system like a large corporation or a country where initially a lot of "small" decisions are made by AIs. Over time, the influence of humans on this system might diminish and the trend might become irreversible.
Currently, no individual and no group, (organization, country...) on the planet has anywhere close to complete control over humanity. Further, even if some country managed to conquer everything, it could not hope to maintain that power indefinitely. An immortal system capable of complete surveillance, however, may be able to maintain power. It's a new thing, we don't know. "Sentience" doesn't matter one bit for any of this.
Such a system might take centuries to form or it might go quickly. Humans might also go extinct before something like this comes about. However, that doesn't mean people who think about such possibilities are stupid.
> That entity need not be a single AI. It could be a system like a large corporation
This is not a new problem, then. Let's tackle the issue of corporations, rather than chase an AI boogeyman that doesn't fundamentally change anything.
Look at oil companies, for example. They have humans in the loop at every level, and yet those humans do little to prevent profit incentives from leading them to destroy the planet. A broken reward function is a broken reward function, AI-assisted or not.
An organization's policies are still implemented and maintained by humans. No matter how powerful, the power is transient. You have churn, corruption, incomplete knowledge transfer, etc. AI systems in effective leadership could be able to maintain goals and accumulate power. That's what's new.
I argue this already happened. Facebook Inc (therefore Zuck personally) in 2015 effectively wielded total control of the world. Adjusting weights in ML systems there could determine who would be President of the United States.
Only when that power became clear to the common man, and to Congress, did any effort to rein it in take place. That fear is why their Stablecoin effort was completely destroyed by the US Government. Too much power concentration. It's why Jack Ma was pulled away to some black site and presumably beaten with reeds for a few months.
Effective and manifested power cannot be reined in. A power that is reined in is only as powerful as its reined in form. Most likely, there wasn't any power to start with and it's just a stupid conspiracy theory...
You can imagine yourself or your tribe or your political party to have absolute power. You can imagine Big Brother, or the Illuminati, or the Teletubbies to have power... It's not True! Today, nobody has absolute power! Let's hope we can either keep it that way or share power amongst ourselves.
A "threat to humanity" need not mean "killing all humans next year". It could just mean some entity gaining permanent control. That situation would be irreversible and humanity would be at the mercy of that entity. Chances are, the entity would have emerged victorious from an evolutionary struggle for power and thus would not care about human flourishing.
That entity need not be a single AI. It could be a system like a large corporation or a country where initially a lot of "small" decisions are made by AIs. Over time, the influence of humans on this system might diminish and the trend might become irreversible.
Currently, no individual and no group, (organization, country...) on the planet has anywhere close to complete control over humanity. Further, even if some country managed to conquer everything, it could not hope to maintain that power indefinitely. An immortal system capable of complete surveillance, however, may be able to maintain power. It's a new thing, we don't know. "Sentience" doesn't matter one bit for any of this.
Such a system might take centuries to form or it might go quickly. Humans might also go extinct before something like this comes about. However, that doesn't mean people who think about such possibilities are stupid.