loved your logic example.

I agree with your thinking.

And as a GOFAI guy myself, I agree that symbolic logic will be key to the next level in performance.

My current guess is that some kind of explicit logic engine will be grafted on, or simple trained from scratch. Either way the transformative jump will occur when the agent can consider its own outputs from the perspective of formal symbolic systems and notice when it diverges.

This itself becomes an internal prompt for more generative thinking. (just as it does with people).

this back and forth between the systems (in a way that is very very loosely like the back and forth in a GANS will be what drives these systems to achieve human-like ability to reason logically over arbitrary large logical structures.

.... and alas we simply will not remain in charge once the systems can do it....

p.s. nice summary of your posts too