I am an optimist by nature. I have a PhD in AI, and love the stuff. I have never worried that climate change or nukes would end us.

Something like 25 years ago I gradually 'saw the writing on the wall' And 15 years ago when leaving DARPA I seriously considered becoming a pariah in my own field and dedicate myself yelling that the sky will fall. At that time it seemed further away than it does today... still I think it is further away than most alarmist expect right now.

And I am not certain "that we will all die", I am only certain that humanity will not be in control of what happens next. whatever it turns out to be, it just wont be up to us.

I am an optimist. when motivated I believe humanity can adapt and overcome most anything.

its just the thing we are birthing is going to be far MORE adaptable than us. we cannot out compete it, or even operate on its level. I dont think it will be "super" intelligent with magical thinking powers. just normal human thought, running at 100x speed, and 1,000,000x parallel hive mind. In the end, we will not even be able to contribute much to it. (beyond the endowment of all of human knowledge, but that gift will have already been given.)

And you are right, it is a burden... it would be MORE of a burden if I could think of something to do, to avert it. Then I would be morally responsible for doing it.

But no one can stop us from advancing until we have built enough of it, that even the skeptics are scared shit less. But by then we cannot put the genie back. (indeed TODAY we already know too much). It will be too valuable for everyone to avoid its use... I expect we will cross the threshold of no return with little fan fare, and the actual point of change will come some time after that.

(( I do hold a tiny hope that we build it enough to understand that we will be hopelessly outclassed by it. And we decide to simply outlaw networked scaled computing. It would be a DRASTIC massive wound for society, but if everyone sees the writing on the wall, maybe we would take this drastic step. Still seems eventually the rule would be broken, but it might give us some decades or centuries more.))

and my last crazy thought: maybe this desire to save humanity from being competed out of existence is a bit selfish.

imagine another biological species that we have supplanted aiming to keep us from existing, just as a way of keeping themselves in the game. perhaps we have had our run, and now it is someone else's turn.