LLMs map on to Dan Kahennaman's human Type-I reasoning quite well, and I think they generally out perform humans at type-1 reasoning. e. g. what you can think of in 500 milliseconds.

And they are a very very poor match for Type-II reasoning.

But let's think about how humans perform type-II reasoning:

We are human after all, and type-II reasoning roughly corresponds to our own CONSCIOUS reasoning. What I observe is that my type-II reasoning proceeds thru repeated application of type-I reasoning.

And this repeated process is guided by idea (sub-goals or modifications of my type-1 reasoning over time.) It seems those sub-goals and modifications occur to me in the same way that type-1 reasoning thought occur to me.

So the key additions required for Type-2 reasoning is training feedback mechanisms that can train the LLM to provide its own signals for type-2 reasoning from its Type-1 system.

This probably needs to be overlaid on some kind of recursive goal, or sequencing mechanism so that its type-1 system can trigger built in sequencing engine to then modulate itself.

This is obviously not a solved problem, and it cannot be done with an LLM alone, but it seems that LLMs are a giant step forward. Before LLMs all prior AI was "brittle" it has no way to operate outside of the space it was trained on. LLMs are very very good lateral thinkers. So good that they hallucinate like crazy. but this is exactly what you need in order to have ideas beyond your own data. You grab ideas from some other lateral space and try to apply them. LLMs can do this well.

But they have no reasoning step on top of this, to cull the wheat from the chaff.

I think in under 5 years we will have the big jump that shows how to train the Type-2 signal into a type-1 system. The results will be another quantum jump in performance... specifically in ability to reason. It will be novel module, but I think the leap is smaller than transformers where... actually much smaller.