I have noticed I am more likely to catch a cold if I sleep without a tee shirt. I think I am a heavy sleeper, and can end up getting too cold but not reacting to it.

Still I feel your basic idea is right. So I am otherwise unclothed, and am careful to have layers that let me shed the right amount of coverage at night.