Actually, it is dangerous to have the rear windows down at high speeds with an infant.
Blow in your baby's face and watch what happens. They stop breathing. It's a basic swim teacher skill, and it works every time. Baby lungs don't fight counter pressure. They can breathe in, but are unable to breathe out. So the wind constantly buffetting the baby's face can, in fact, cause the baby to stop breathing. Of course, you're going to ask me to cite this and the simple truth is I can't, and Google did nothing for me just now. Which is why I told you to blow in your baby's face. And if you keep blowing, the baby will continue to hold her breath. Please stop blowing before your baby turns blue.
With an older child, this is less of an issue, but I still would not be willing to have the windows down while the kid was sleeping, or period while the kid was RF. I have my windows down about 1/4 way in the back at times, but normally only when Charlie is screaming and opening the windows calms him down (or drowns him out, whichever the case may be). Unless I can verify that he is breathing (screaming, playing, I can see him, etc), I don't do it. Once he's FF I might be okay with it as long as he is awake, but there really is very little opportunity here to have the A/C off, anyway.